Currently I am developing a game system for an RPG. I'm trying to figure out all the categories of rules.
So far I can think of character creation and advancement, combat... and thats really it. Is most of the content in guide books simple lists and lists of items, spells, places, and people, or am I just missing something?

It depends on how detailed you want your game to be, for example, in DnD there is also encumbrance rules. This tells you how much a character can carry based on their strength score.

So there could be lots of rules you can add, depending on how realistic and 'crunchy' you want your game to be. In my own game, I use tables to help determine material damage, so say a zombie is attempting to get into a building, how high does his strength have to be to punch through the wooden planks that bar the windows? Will the wood eventually break, or will the zombie punch at the planks until it's hands turn to a pulp?

These are the things you can think about. Though too many rules like this, in my own opinion, can bog down a game. So it's up to you to decide what's important for your style of game, some rules might be more important to include than others.

What I'm working for is almost rule heavy but in a template design that allows for a lot of growth. When I have worked on games before I always get through character creation and combat, with some condition rules, but then I wonder where to go. I think it is the flavor text and world filler I never get to.

A good, clear explanation of the game's mechanics is a must-have, for me.  It took me forever to get into Shadowrun because the way they wrote up the mechanics of the game in the early editions was hard for me to grasp.  Once I sat down with some friends (with more patience than I have) who'd deciphered what the rules actually meant, and started playing, I had a lot of fun.  But for the first year and a half, I basically used the rule book as setting fodder and ran my game with the D6 rules (because I, and everyone I was playing with, already understood those).  And I don't just mean combat...

Examples of how to apply skills, limitations on how many different skills you can apply to a combat scenario at any one time, how healing/recovery works...these are all vital to any game (that's where playtesting becomes a must...you may think you've got a rule for every imaginable situation, but I guarantee you, if you have ten other people playtest the game, they will find at least one major hole, each, that you never even thought about...).  If you're making the system to work with a specific setting, then you should have at least some basics of the setting figured out for players to use as reference material.  Available equipment and how it affects the game (damage from weapons and any prerequisite skills to use them effectively, bonuses from various tools, for which activities, how a first-aid kit or its analog will affect healing rules, etc) is also pretty critical, as well.

A general layout (not necessarily in this order or using all of the pieces) is

• Introduction to the RPG (what is an RPG, what is the point of *this* one, brief description of the setting/genre)
• Glossary
• Setting Info & Premise
• Character Creation
• skills and how to use them (inc. item crafting and complex actions)
• Classes/Abilities/Spells
• Combat (inc. complex penalties, bonuses, normal actions in midst of combat)
• Misc rules about healing, poisoning, stress, sanity, whatever
• equipment lists
• bestiary
• INDEX

It's pretty important to know what you want the rules to model.

Are you trying to make a generic, fits anything system (like GURPS) or something very specific and focussed?

Do you want to simulate reality or pulp fiction or action cinema?

Is there a specific genre? medieval fantasy combat is different from modern military combat is different from mystery/detective is different from horror.

All of these will impact what you want to include in your system in terms of skills, healing rates, equipment, encumbrance, etc.

Playtest, playtest, playtest, and don't even bother waiting for a complete draft of the rules or a full party of players to start playtesting.

I am making my own system as well, and even though none of my test games actually started gameplay (yet), the questions and feedback (mostly questions though) brought to light so much about things I had forgotten, or things I thought were clear but weren't, or even things that turn off players or confuse them.

I.E. I had originally made humans strong (a bonus to strength) because humans are, in real life, strong for their weight, but during playtesting it became clear that it is better for humans to be considered the norm (mechanics wise), even when they are not.

Also, consider for a moment what you want to support in your players. Classes make creation easier or faster, and give inspiration to those that need it, but they also limit creative options. I.E. I want a fighter who attacks with swords and bows, but uses magic in a support capacity, invisibility, speed boost, walk on walls, etc, but DnD and PF make it very hard to make the concept work.

So classes don't support interesting creative characters, but they do help if you want to support a certain combat or dungeon delving style requiring certain abilities to be always covered by the group (aka the style of play that requires those people asking about what role they need to play)

Some good advice is to decide on what you want the players to do in the game, then to make mechanics that enable and reward them for doing that.

For example, I want to support creative thinking and creative characters, so I have no classes in my system, and abilities tend to be open-ended rather then specific, all to enable and support creative use, but it comes at the cost of being easily gamed during combat if the GM isn't careful. Thus the game will hopefully do well in balanced gamelplay, but not so much for "kick in the door and hulk smash kill everything!" style of play.

@facemaker

Thanks, so I need a full concept of what players are able/allowed to do. I liked how shadowrun explains a lot of situational modifiers.

@ Tileira
Thanks, that is exactly a list I wanted.

@ Gaffer

I'm trying to go semi realistic. Encumberance is slightly unreal in that I can fit 3 long swords in my pocket, but I like that. Fantasy I always like easy to go beyond the mundane.

@ DarkLightHitomi

And that is the trouble I face is finding the people for it.

Also I like loose class structuring.

As far as classes go I like the names. I always like adding more and more to the list. One Idea I've been fiddling with is using perks, races, and classes to describe a character instead of skills and attributes.

I think it heavily depends on what kind of game you are planning to make. In a roleplay which is supposed to be about the modern world you will need other rules in comparison to a RPG set in a medieval fantasy world.

I would definitely add some skills and rules on how to obtain them.

I've found that I, personally, prefer games where character advancement is done via buying skills and abilities, without any kind of 'level' system and not hampered by 'class' prerequisites.  I know some people really like that aspect of D&D, and how it kind of focuses and streamlines the game...it's actually one of my least favorite things, and one of the reasons I've always enjoyed the D6 Star Wars rules...you may be playing a bounty hunter, for instance, but it's entirely up to your discretion when it comes to how that bounty hunter works.  Nothing in the rules tells you what kind of skills you should get as a bounty hunter, and how often you should raise them, and there's no assumption that just because you've been playing a bounty hunter for a while, you'd gain significant proficiency with a blaster or vibroblade or whatever...

I know some people really dislike systems like that...but since I'm very much a Jack-of-All-Trades in reality, I like systems that allow me to build characters along the same lines.

A lot of very usefull stuff has been said! I think that it is important to see what kind of game you want. Setting and tone have to do with it, but also the goal of the game. Is it a contest of players trying to beat the GM? Is it players solving problems in creative ways? Is it story driven? Is it character driven? Does it just try to emulate or simulate the real world, perhaps with some extra special things, but everything needs to be realistic or will it be more like reality needs to fit the plot. (realistically the character should die now, but he can't because he's the hero!)

Every different approach needs carefull consideration. It is very hard to make a game that can encompass all, and even if you do, it will be mediocre in it. Find out the goal of the game (beating enemies, showing of an uber character, telling an interesting story, exploring life in the middle ages or something, surviving in a hostile environment, or just having a great time, rolling as much dice as possible.)

Try to find a reward mechanic that rewards the desired player actions for the goal of the game, and rewards it in a way that supports the desired play. I once read a great article about reward systems, but I don't have the time to go try and find it right now. Anyways, find the goal or purpose of your game and try to build it around that.

And so I have found the holy grail question I've been looking for. Thanks lots. I'm gonna go work with that now.