Virgin to 1st reload you'll usually see a slight velocity change. 10-15fps on the average. 2nd, 3rd, 4th firing you're not likely to see any change. 4th-7th firing gets into the realm where the neck is work hardened and you may or may not see velocity ES climb up a bit, and you'll start getting neck splits if you don't anneal.
Whether you anneal every firing or every other or every third firing (provided you get it hot enough long enough, but not too hot for too long... which is a whole 'nother thread...) isn't really going to show much of any difference you'll be able to notice/measure for velocity spreads or accuracy.
Annealing is not likely to shrink ES/SD unless your necks are already very hard (typically that 4th-6th firing zone). Especially unlikely to see a change vs virgin cases.
Whether or not it's worth it depends on if you're going to run into other issues first. If you're running loads hot enough (or have a semi-auto gassed up enough) to lose primer pockets, or if you bump the headspace .005-.010" every firing and you're going to get head separations, there's little point in annealing. However, with mild loads and minimal sizing you can get 20-40 firings in bolt guns, and 10+ in gas guns. YMMV.