Fill material and fill volume both play a big role. Bag shape as well. A soup can shape is my favorite by far. Are you properly squeezing the bag to firm everything up before the shot? The rifle should be able to freely slide backwards and forwards during recoil. I’ve used everything from sand to plastic BBs. Rice is one of my favorite cheap light fills if the bag won’t see water. For a bag that might get wet I have some with whatever light fill Armageddon Gear is using. Sand is by far the best stuff if you don’t mind the weight.
I also use a Protektor leather bag when shooting off of a Seb Mini rest. They are extremely firm and in that setup all adjustments are done with the joystick on the rest.