If you look at all of the 6x5's the group size from 50 yards to 100 yards usually goes up by a factor of 3, not 2. People who are putting in .5-.6" groups at 100 yards, are putting in .172-.2's at 50 yards.
If you're doing around .37-.4, then you should be expecting about 1.1 - 1.25" at 100 yards. So I don't think it's falling apart, it's just the barrel/shooter. So you should be able to verify if you get a new barrel that if you get down to .25 at 50 yards, then .75 would be generally your 100 yards.