A few years back, I had a post on another forum asking to diagnose my issue with a 6.8 SPC upper I had built. Like you mention, I was able to shoot a couple tight shots, then the group would open up. I asked a fellow good shooter to get behind the gun and he put 5 shots down range that measured just over 1/2 inch.
Sometimes we are too smart for our own good; we so badly want a one-hole group and we pay attention to attaining it......instead of sights, trigger, follow-through. That was my problem and my experience.
Lastly, you might put a paper target at 200 or 300 yards or beyond (non-Shoot & See), and focus on executing 5 good shots. You won't be able to see your results and they may surprise you.