As long as the firing pin is not sharp, jagged, oversprung, etc.. The crush that's happening between the primer cup and the anvil should be extremely similar between large rifle and small rifle primers. If there is a difference in cup wall thickness, it's not much, so the area directly around the firing pin impact isn't that different.
What DOES cause the firing pin crater to detach is a very loose fit between a small firing pin and a large hole in the bolt face which allows the primer cup material to shear away into the FP hole. Separate issue from simply the diameter of firing pin.
Smaller diameter properly fit should not cause any problems shooting LRP or SRP.