What happens when a bullet is over stabilized?
There isn't a one size fits all answer to this question. Just too many variables to state anything in certain terms, but we can talk generalizations. In general we run into short range issues with the bullets coming apart and adverse effects on short range accuracy most often.
The most obvious to observe, is that bullet jacket can not take the rotational forces being exerted on it and comes apart. Once the jacket separates from the core it looks like shrapnel flying off the bullet and it's anyone guess where any of the pieces will go, including the core. Solid bullets, as Frank mentions, eliminate this as there is no thin jacket to break apart.
The second thing is a little technical with effects of yaw, nutation and precession. So we will gloss over all the math and physics no one wants to do.
The easy way to understand this is to think of the tire on your car. Perfectly balancing tires and rims in manufacturing is nearly impossible. Thus, balancing weights are added to the rim when the tire is mounted on the rim to balance everything. When you drive down the road at 35 and at 65mph with a balanced tire/rim it rides completely smooth. Now if you take the balancing weights off the rim you will likely still have a smooth ride at 35mph, but as you spin the tire/rim faster to 65mph you will start to feel a shake.
Bullets are effected by this same effect of not spinning perfectly smooth due to not being perfectly weighted. This shows up as decreased shot to shot accuracy at close ranges. As the bullet slows down at longer ranges just like with the car example the unbalanced effect on accuracy becomes minimal to imperceptible. The car example makes it easy to understand, but it's not quite perfect as bullets don't fly in a perfectly straight line as shown in below picture from a University of Utah Library ballistic article. Bullets actually experience center of gravity shifts during flight that car tire/rim are not subject to experiencing. Yet, it remains a good starting point without getting into heavy reading and lots of math.
Now recently there is some interesting hypothesis regarding "hyperstabilized" bullets, though it is backed by some very intelligent and highly regarded names, that's still largely academic at this point in time. It essentially states that it is possible to eliminate yaw(see blow picture) and thus minimize to eliminate the effects of drag from this movement during flight. In turn this will dramatically increase the BC of a bullet within certain flight parameters. Time and further real world testing, such as Frank's fast gain twist barrels, is still needed to fully validate such claims.
