My understanding is that the 168MK is on the ragged edge of dynamic stability, and in some cases can be neutrally stable, where the yaw increases and stays put. In other words, they won't necessarily tumble, but it doesn't take much to make them go sideways. Spinning it faster will help with that, but only a very small amount. Once a bullet is gyroscopically stable, increasing spin provides rapidly diminishing returns. Bullet design plays a huge role, and the 168SMK is unfortunately not a terribly good design for long range. Trying to shoot these past 800 yards is something people may want to do for fun, but it's really impractical, as there are much better bullets. A mile seems like a huge stretch, although I've never done it myself.
The reason JBM said your bullets would be stable is that it calculates an approximation gyroscopic stability, not dynamic stability. Dynamic stability is not easy to calculate because it requires lots of very difficult to obtain data specific to the bullet. For a bullet to be stable, it must be gyroscopically stable and dynamically stable. As you've seen, the 168 goes wonky on the dynamic stability when you push the range.
If you have a better bullet that is stable through to the target, then increasing spin will not do much of anything.
Unbalanced bullets are another problem entirely (well, they're somewhat related in that they cause bullet yaw). An off balance bullet will want to tip while in the bore, but it can't because the bore constrains it. As soon as that bullet leaves the rifling, it will do what it wants, which will result dispersion. The faster you spin it, the bigger this effect will be. This is why (almost) everyone wants to shoot the slowest twist they can get away with. I have a .30BR with a 1:18 twist. It shoots great, but only with bullets under 125 grains or so ( I know it works with 118's, not sure what the upper limit is). It cannot stabilize the heavier ones.
