It's bad correlation to refer annealing to increasing accuracy. Is more of allowing better consistent case to case which leads to accuracy.
You anneal so you reverse the effects of work hardening the case. By doing that it becomes softer, more malleable and after sizing, less springback.
Think of work hardening like tight muscles. You take a warm bath to relax muscles. That's annealing.
Theres times when you necessarily don't need to and I'm going on a limb here saying this but if you neck turn cases and /or have a very specific reloading set up. In the end you have your cases the same and don't necessarily need to anneal.
Long story short, yes you should anneal. There's multiple benefits to it, jury is out whether you need to do it every firing, 2 firings, 3'firings. No ones proven or disproven one way or the other.
On a a side note if you can't afford an annealer, killshot (look for his sticky above) offers annealing services and it's VERY cheap.
A really good option for those who can't afford annealing machines and want someone with the know how to get it done right.