If it's too high for comfort, and you have room to go down, I don't see why you wouldn't try and go lower.
Height over bore impacts rifle fit and the magnitude of cant influence. If you are level and have great rifle fit & sight picture, a tall height over bore is no issue.