Those hardness numbers don't really mean what you are thinking in this context.
a) brass work hardens, and the surface will become harder from peening (whether from stainless pins, tumbling impacts, shot peening, whatever)
and
b) the hardness difference between stainless and brass is not the only measure, or even the most important measure, of how much the stainless might damage the brass. Sharpness of any edges on the pins, and amount of any abrasive material (including dirt and primer residue) involved carry more weight.
Along with that, generally speaking a softer material can often wear through a harder material if any abrasive material is involved. The abrasive (again, could be dirt/dust, primer residue, etc) can imbed into the softer material so that it wears into the harder material. I've seen this a number of times in my professional career with various types of durability tests, where even a plastic wire loom can wear into/through steel and aluminum parts. Keep in mind this is looking at wear/abrasion, not impact damage where the roles between hard/soft materials align more with what most people expect.