I had a similar accuracy problem with my V-22 that I've fortunately corrected. Before I get into the details I want to set the stage.
There's two schools of thought about rimfire barrels, the first is that they don't need cleaning very often or not at all, the second is to shoot their best they need to be as clean as the you can get them. The positional, PRS style shooters, as a general rule fall into the first group, the rimfire benchrest competitors into the second.
I began shooting my V-22 with very little regard to cleaning the barrel ... after about a year of modest maintenance, much like you described, accuracy fell off dramatically. I'd estimate I had four or five thousand rounds down the barrel when accuracy dropped off. I talked to Paul at VGW and considered sending the rifle back to the factory but before I went to that extreme I convinced myself that the drop off was in the barrel and I needed to remove the dreaded "carbon ring". At that point I hadn't added a bore scope to my kit bag.
Here's what I did to solve my problem ... a thorough scrubbing with J-B Clean concentrating on the chamber but treating the entire barrel. Before I used the J-B Clean I used my standard protocol - a soaked Bore Tech Rimfire Blend patch to wet the barrel followed by a brass brush cycled several (5 to 7) times back and forth through the barrel and then wet patches until they come out clean.
Then I used the J-B Clean and followed again with my standard protocol ... I was amazed how dirty the patches were that came out of the barrel. I repeated the standard cleaning once more to make sure I removed any residual J-B Clean and off to the range I went.
My range-day results were far better than I'd seen over the previous three months and I was encouraged that I'd identified the problem and had a solution that would work. By the way, my V-22 wasn't nearly as fun to shoot when the accuracy disappeared - ain't that the reason we buy these rifles anyway?
A couple more thoughts - there's several similar cleaners you can substitute for J-B Clean. Bore Tech makes C4 Carbon Remover and Birchwood Casey makes a Lead Remover and Polishing Cloth. Both are available through Amazon. A fellow Vudoo owner uses the Bore Tech product with great results and it may very well become part of my standard cleaning protocol.
My second thought is that you might consider adding a bore scope to your cleaning protocol. I bought one from Teslong that works very well on rimfire barrels. I scope the bore prior to and after each cleaning to ensure that I've returned the barrel to pristine condition. You just can't see what's going on in your barrel without a scope.
Like I said at the beginning of this post there's two schools of thought on barrel cleaning ... obviously I'm in the benhrest shooter's camp. My shooting is at three yardages, 50, 100 and 200. I participate in monthly competitions at these distances and keep records of my target scores, round count and barrel cleaning. The data I've accumulated has convinced me that a clean barrel is a happy barrel. Sure, you'll have to shoot a few rounds (for me about three) to settle the barrel but then you're back in business with the Vudoo accuracy that always puts a smile on your face.
Hope this helps you and others with similar experiences.