I am having similar result with 308 and I am using lee collet neck die...
Has mostly started since brass has hardened and needs annealing but 001 to 002 shoot great the 004 to 006 shoot a lot worse, but I think its due also to neck tension being all over the shop...
I have not turned necks so...