Not sure which action you have, but in the Gen 1, as long as the magazine was inserted, you'll never feel the ejector. A long time ago, we did receive ejectors that were a tad too tall, and a few swipes with a #2 file to remove .01" to .015" resolved any contact issues. For the masses: This is no longer the case, so please don't call the shop and tell them I said your ejector needs to be replaced or repaired.
If you have a Gen 2, I'd be incredibly surprised if you're having this issue, but in the event you are, the fix is the same....a few swipes with a #2 file on the very top of the ejector, the file should not touch anywhere else on the ejector.
MB