Here's my opinion on this. I am a rifleman in the USMC, and I've ran through my fair share of chest rigs, vests, plate carriers etc. In my eyes, if you don't intend to wear body armor, just put it all on a belt.
Find a nice padded MOLLE belt and maybe a set of suspenders, put a double-double mag pouch on it, and then whatever else you want (pistol holster/mags, knife, first aid kit, dump pouch, GP pouch etc.). My issued flak jacket has a pouch on it for all of the 10 fucking pieces of optics/nav gear the Corps issues to me, 1 mag pouch, and 2 grenade pouches on it. That is it. Everything else goes on a belt or a pack/camelback.
You want the weight on your hips, not your shoulders. Your torso moves when you move (walk, run, whatever), and all that weight moves with it. Not a big deal for a shot duration, but over the long haul your shoulders and back will start to fatigue. Chest rigs and vests tend to hold in a lot of body heat, too. Your hips hardly move at all, so you're not 'flinging' weight with every step, just to have it fight you. I think they offer a lot more personal mobility in terms of bending/moving. Belts are also 20x better in the prone than lying down with full mag pouches on your ribs.
That's my take on it, anyway. If you're going to have a flak or plate carrier with sappis anyways, go ahead and throw some stuff on it, but if you don't need shit on your chest, don't put it there.