You're explaining pretty much every rifle ever created.
There is a reason you track your clean cold bore as well as your fouled cold bore and where they hit in relation to the next shots in your string.
The reason this is happening is (and you'll get 20 different explanations of this) the round going down the barrel with all of the fouling along with it are filling the microscopic gaps in the metal inside the bore. It's uniforming the bore more as well as creating a more consistent bearing surface inside the barrel.
Think of it this way, you clean all the shit out of your barrel. Now you shoot it once. The second shot will now experience a slightly different condition in the barrel because of the first shot. Continuing this has diminishing returns as you go and for your barrel that seems to be at around 10-15 rounds and will stay in that range until it changes too much; maybe at 400-500 rounds. This is why people usually laugh at those that shoot 10 rounds and then clean their barrel each time.
I'm sure there's a much better and scientific answer for you, but that's the gist of it.