The tire size by itself isn't quite as important as the relationship between the front and rear tires.
Front tires should be slightly larger, an inch or two, than the rear.
If you have more tire on the rear, then you'll have bad understeer and steering input will increase.
18-20" on the rear is ideal for most all environments, 21-22" is great for desert X/C but you've still got to find a front that's an inch or two larger.
Anything less than 18" is worthless unless you're running on pavement.
Wider is better in all situations for the rear, on the front there's several considerations depending on terrain and type of riding.
Last consideration is sidewall...the more sidewall you have, the more it will cushion the bike and conform to terrain (more traction).
The drawbacks...tires will wander and handling gets a little sloppier.
Flipside, less sidewall has less traction and a harsher ride, but handling on smooth surfaces improves.