I'm an older guy and not big and I'm no tough guy. I'm was long time clay target shooter but now 2 years new to scoped rifles.
While I have an 19 lbs 6.5 CM, I also shoot a 10 lbs scoped Rem 700 in .308 with a Harris bipod (shhh....don't tell Frank I'm happy with this bipod! haha).
I am starting to shoot prone more, but have most often shot off of a bench or window sill in a box blind (and my lumbar loves benches! haha).
I have also have never had a 20 mph wind move my point of aim. IMO, there just isn't enough surface area in the profile of a rifle to provide sufficient force that it would move my point of aim around. No way.
And, I have shot tournament skeet which is all off-hand, of course, in hideous winds (and yeah, it sucks for lousy scores) but never, ever, felt that the wind was moving my gun.
My only thought, and this comes from skeet also, is that the wind can get in you mind and cause you to think you need to compensate somehow, change something someway, and end up chasing the cross hairs around. Maybe??
And, my only suggestion is to try to shoot in the wind EXACTLY like you would in dead calm....I don't mean to not hold for the wind, I mean get it out of your head and set up and execute as if you were at an indoor range with zero wind. Don't tense more, don't tense less, just do in the wind what you do in no wind.
Thoughts from a relatively new guy