I agree with winniethepooh. If your barrel is not free floated that could also be a problem.
My rifles will also shoot a little higher from a clean bore with oil in it. After a fouling shot or two or thee or four, it will settle back down to zero.
If I remember to do it, I will take a small bottle of acetone with me prior to varmint hunting. Then right before going to the field, I will swab the bore out with a patch soaked with acetone, to get rid of any oil. That doesn't quite replicate the conditions after a fouling shot is taken but at least the oil is removed.
I've also done that prior to checking the zero at the range. It's not quite like a fouling shot but gets the oil out. Remember that after the first shot, all the oil is blown out of the bore. With the oil in the bore, the friction is less and velocity is greater meaning that the point of impact is higher on the first shot. At least that's my theory on one factor that affects the first round shot. However, it usually doesn't take 20 rounds for my rifles to return to zero.
Another factor is how you have your rifle zeroed; on sand bags, a bipod, sitting at a bench or prone. I try to zero my rifles the way I hunt with them; on the ground, prone and on the bipods. If you change the way you are checking the zero every time you shoot, you will definitely notice a change in the point of impact. For me, I notice a big change between shooting from a bench versus shooting on the ground in the prone position with bipods.