Re: factory barrel
What stock is the SPS-V in? What optic are you using?
Cutting, recrowning and rechambering the barrel will probably have a positive effect on accuracy. Remnington's factory chambers are pretty loose and a shorter stout barrel tends to be more stiff than a long one so there may be some accuracy gain there. Blueprinting the action can help if you have the barrel cut back and rechambered at the same time. If the bore of the barrel is crap though cutting the barrel back, recrowning, rechambering and blueprinting won't help much.
Let's look at the cost of cutting, rechambering and recrowning, you are probably at somewhere in the neighborhood of $200-$250. A new barrel in the length, contour and with the crown style of you choice from McGowan and short chambered is $300 or so and as mentioned above you can pick up a Shilen short chambered and otherwise ready to spin on from Brownells for $275 or so.
With a decent stock (B&C $200), a blueprinted action ($200 or so) and a new barrel you are at around $700 and shoot .5 MOA. For another $125 put a Timney trigger on it and have some fun. These are all things yo can do progressively, you don't have to dump all the money at once.