I had this problem for years and could not figure it out. Drove me absolutely nuts. Said I was done with AR's a million times over. I have an SLR hand guard and an SLR adjustable gas block. It looked like it had plenty of clearance, like .100" at the closest point. I also just assumed SLR would make their gas blocks to fit under their rails. Somebody made the suggestion I just try a smaller block even though I was sure it had plenty of room. So I swapped it out for a standard low pro gas block and it was a miracle. For the first time ever my ar15 actually shoots like a bolt gun. It has a cradack barrel on it and it shoots under an moa every time now. No flyers, no weird shit. After about 3 years of trying every ar part available and every bullet powder primer seating depth combination, this was the fix. Now every bullet shoots pretty decent out of my rifle. I have a buddy with the same exact set up except a WOA barrel. It has always shot about 2 moa or so. I swapped the gas block, threw some of my random hand loads in his rifle, and put 5 rounds into one little hole. Every group after that was right on with his rifle as well. The zero shifting, bad groups, unexplained flyers are now all gone.
What is your hand guard and gas block set up?