The purpose of using an abrasive on the chamber walls after reaming is not to smooth out the surface, just the opposite - to give it some "tooth" for the brass case to engage when under full pressure as it momentarily expands. In Gordy Gritter's book he recommends nothing finer than 320 grit. The rationale is if the chamber wall is too smooth, more force is transmitted to the bolt lug/lug abutment under pressure. At 50k--65K psi, this bolt thrust over time will promote greater wear.
Edit - I should add that there is disagreement as to whether chamber wall finish, bolt thrust, or even brass flow merits scuffing the chamber wall.