I'll go against the grain and say I do what the OP is asking about. I switch my ZCO between my centerfire and rimfire because the scope is that much better than my backup and it takes a lot (A LOT) of rounds rezeroing to pay for another ZCO.
It is inconvenient, especially going from the rimfire to centerfire, but it's much more affordable than another alpha tier tube. My POI shift has been enough to take me off a standard 12-14" target at 100 yds though, ymmv. My go-to recommendation for zeroing a CF rifle is always to put one hole in paper at 25 yards, then adjust the POA to the same height over POI as your sight-over-bore dimension. That'll get you very close to zero at 100 yds in one shot, then you can go get a true 100 yd zero.
Sadly, don't always have access to a 25-yd line, so it's not a perfect solution.