You're right... and I was going to suggest that. Probably the torque on the action wasn't the same when you got your zero, as it was after you removed the action and re-torqued things.
Different shooting positions (bench versus prone, etc) can also affect POI and zero. Too, when a scope is...