It is a tool, and it is a very adaptable tool at that, so much so I have more than a few. Set it up how it works for you. The simpler the better for sure, but if you need the rail it is indispensable. Having used it a lot, I kind of know what to do with it and how to make it work for me. I guess I'm a fan of the rails and add on junk, guilty as charged. But I do find the KAC rails, whether you use them or just cover them all up, and FF or not (but if you spend the money, just get the FF) are much more stable than the regular plastic handguards that move around. They vent heat better too, FF will tighten your groups, and with all four long rail panels clipped in, it is just a regular handguard with future potential should you need a bipod, etc.
Personally, I prefer my basic go-to M4 to be setup a lot like I had it in the army but with some minor tweaks, with ACOG, KAC rail and rear 600m flip (fixed front) surefire grip light, sling stud, PEQ, AAC suppressor mount, and a grip with an extra bolt and pin in it and the SOPMOD stock full of batteries and other small parts for the rifle. This is comfortable, familiar for me, and the only one I have fully decked out. But I also have training with and know how to use all this stuff and make it work for me. I kept a bipod on mine in the army most of the time, but I have others more suited for longer range shooting now. My wife has an M4 exactly like mine but with just the EOTech and a grip and a sling with an old Surefire weapon light mounted way out front on the side, the rest is covered up so you can still hold it normally. That's how she likes it, that's the limit of what she needs and for next to the bed, what else DOES she need?
On my shooters, they still have KAC rails, but FF, and they are mostly covered up too except for the spot for the sling stud and the bipod mount. One even has a 1907 sling along with a USO SN3, but short of that, it is a plain 20" A4 style 6.5mm Grendel (but with really good guts inside).
So to each his own... But nobody should be giving you shit for choosing a simple weapon. Everyone has personal preferences, and a weapon is a serious item, it is what you defend yourself with. So if your training and comfort lies with a light, plain AR with just the ACOG, then so be it. If you feel more comfortable adding weight in order to get other tools that you have trained with that make you feel more comfortable, well, then so be it too.
Don't feel compelled to add shit on. If you don't feel the need to, then it is likely you don't need it in the first place. Never trained with it, no need for it, all it will do is make your rifle feel foreign and unfamiliar.
Some people criticize for the sake of criticizing someone, and non-constructively at that, the worst kind, "you should have done this instead" or "that looks tacticool/you're a mall ninja" or "I can't see the point of this or that". That is all garbage. Constructive criticism might be to discuss the application and use of an item so as to suggest a better alternative. Such as what I mentioned in the first paragraph. Know to look for the difference if other's opinions bother you.
Anyone giving you shit because you don't have rails and add-ons likely is from the mall ninja mindset. You don't want to take advice from a mall ninja do you? Didn't think so.
And look at what the Marines use: M16A4 with no intention on changing. Most have rails covered up, with ACOGs, some have the Harris bipod, very few have grips from pictures I've seen. Most use them just for the bipod or the laser it seems, if they have one. Simple seems to be working for them. Granted they have the rails, but like I said, they can offer a good stable handguard covered up (and potential should you need them) if you don't need the rails.