I definitely understand wanting to trim based off the shoulder, as that way no mandrel rides around in the neck, but I have had problems with that method given that the cases need to have their shoulders bumped back uniformly to the same datum for the shoulder referenced trim to yield consistent results. If you're neck sizing, do not expect consistency from shoulder referenced trimming.
Now, if you're full length sizing and bumping the shoulders back such that they're all uniform AND you're trimming from the case bottom to the neck rather than referencing the shoulder, I'd think .004 is a bit on the high side. I use a RCBS electric case trimmer and am +/- .001" usually which I find absolutely astounding for such a rickety machine. I would look to tighten up all the fasteners on your Lyman to make sure there's not some variance there. Also, keep in mind that measuring a case is still a motherfucker with calipers, especially if they're not the highest quality. I would pay as much attention to the method you use your measurement instruments as you do to how to trim your cases.
All else fails and you still have a tolerance of +/- .002 (which is what a variance of .004 is), I think you're safe. Every bit matters and granted I don't know you or your shooting skill, but I have a hard time believing that you're going to see a few thousandths variation in neck length on paper, especially if your neck tension is nice and light.