There are eight things that contribute to your sort of problem. Assuming you are shooting from right shoulder and you are right eye dominant, you should look to see if these relationships are consistent or out of proper alignment: high or muscled right shoulder, movement of the right elbow from shot to shot, high vertical position, no stability in position or a muscled position (having not properly adjusted NPA), jerking trigger, varying head pressure from shot to shot, failure to maintain alignment, and/or sight picture. Of course, since all shooting is at the firearm only you can discern what listed here is part of your problem. Proper call/strike analysis will help you understand work in progress when cognizant of the five factors of a steady position. Also, be sure to shoulder the rifle and align sight to eye, cognizant of the five factors of a steady position but without consideration of the target. This makes adjustment of NPA to get the desired hold much easier. And, remember, wind can also be a factor with about 1.5 MOA of drift shooting at 100 yards with a high velocity rifle in a 15 mph cross wind. BTW, the five factors of a steady position describe the contact between shooter, gun, and ground. These are: non firing hand, firing hand, butt to shoulder, stock weld, and elbows. One more thing, your rifle could require some fresh bedding. At any rate, I think I've covered all of the source of error possibilities here.