Normal Map Workflow
This tutorial will explain my workflow when making a normal map for a character. I'll be using 3DSMax though most of it will transfer to any program. As with everything I write, take it with a grain of salt, compare it to what you know, and figure out what you want to use from it for yourself.
First I'm going to explain what a normal map is and does, from an artists perspective. Imagine a styrofoam head that is faceted and chunky. This is like our low poly model. Then imagine coating it in clay and sculpting a much more detailed face on top of it. Then you slit this clay from the back, and pry it off. Lay it flat on the table, and you will see that it is thick in some places, and thin in others where the edges of the chunky head (the low poly model) came close to the surface. This layer of clay is what gives the definition to the sculpture, whereas it was featureless and flat before the clay was added. This is what a normal map does, though it is virtual and doesn't change the actual silhouette. It tells the engine how to display the light hitting the object. Current hardware can only handle a certain number of polygons still, but normal maps give us more detail without actually stressing the processors as much as trying to render the high poly models that the normal map comes from.
Next I'll explain how the "bake" works. The low poly casts "rays" out from it's surface, using it's smoothing groups to determine the direction the rays are cast. You can enhance and tweak the rays in some programs by using a "cage" which I'll explain further later on. If you apply smoothing groups to a model, it will not only display a razor sharp crease in the model in game, going against the look of the other defined edges in the normal map, but the "bake" will also miss portions of the high poly. This image should help explain.
The first thing I do when making an asset that will have a normal map, is to make a "sketch" model. I make this model quickly, trying to hit my target polycount for the in game model. I pay attention to silhouette and meshflow as well. This is so that I can know how much detail my low poly will "hold". If you make a high poly model, without any regard to the polycount you have for the low poly version, you might find that you made it too detailed, with too many things breaking the silhouette, and then you'll have to go and delete portions of your high poly, wasting valuable time. Normal maps work best on large smooth planar surfaces. The rounder and smoother your low poly, the better the normal maps will catch and hold. Whenever you have to make too many quick changes in surface direction, the normal map won't work as well. I try to keep my low poly one solid mesh, with one smoothing group. The reasons I make my meshes watertight, is that when you have intersecting elements, the normals don't average. This means when the normal map displays it won't average either, and you will have a hard intersect. I didn't save that state of this model, so imagine that this model is rougher and doesn't match up with the high poly so well.
Layers are extremely important when creating normal maps in max. These are the most commonly used buttons for my workflow. I encourage you to learn the other buttons as well, but this should give you a crash course in the layer panel. In order to bring this layer manager up, click this button.
I keep my sketch model on a layer called "low". I make a new layer named after whatever part I'm working on. In this case it was "body" and I merged in a body I'd made before and started altering it to fit this project. I'm not going to cover high poly asset creation here, I'm going to assume you've made your high poly in your package of choice, be it zbrush, XSI, Modo, or others, and merged it back into max. Try to keep all your chunks as organized as possible on different layers. This is especially important if you don't have a great video card, as you can speed up your viewport manipulation by only having one layer visible at a time.
Now comes time to tighten up your sketch model to match up with the high poly. I like to put the low poly on X-ray mode (alt+x) so I can see the high poly inside it. To speed up your view, only show wireframe on selected object, as displaying the wireframe on your high poly slows the scene down a lot. To do this, right click perspective in the upper right corner of your viewport and go to Configure at the bottom. Then in the Rendering Method tab, tick the display selected with edged faces button. This way when you have edged faces turned off, whatever you select still gets edged faces. Try to match the surface of your low poly as closely as possible to the high poly. The goal is to capture the silhouette and volume of the high poly. Pay particular attention to parts that break the silhouette. There is a script on Scriptspot called "Tim's Scripts" that has a conform macroscript in it. If you have a solid mesh as your high poly, you can use this script to pop the verticies of your low poly to the surface of your high poly.