When asking what is the most powerful round you can load in the AR-10, the real question needs to be where are you measuring that power?
At the muzzle or downrange and at what ranges?
Your maximum bore diameter cartridges with the most propellant, chamber pressure, and projectile weight will have the most muzzle energy, but due to low BCs, will bleed momentum faster than other bullets with better SD/BC. This would cover your .40-.475 bores. Think 450 Marlin (Accuracy Systems does this build in AR-10s), 45 Raptor, and 475 Bishop Short Magnum.
450 Marlin
475 Bishop Short Magnum
Your mid-bore size (.338-.375) high SD bullets fired from the same propellant mass will provide better momentum at medium to long range. 325 WSM, 338 Federal, 358 Winchester, 375 Raptor, 375 Bishop.
Your smaller bore, high BC and lighter cartridges fired with similar propellant mass will maintain momentum at longer ranges, but will have less energy on-target at closer, typical hunting ranges, though they will be more than enough. This will cover your 270, 7mm, and 30 bores like 6.8 Western, 7 SAUM, 7 WSM, 300 SAUM, 300 WSM.
Normally when we’re talking about big bores or high muzzle energy and closer range performance, it’s about some type of hunting setting based on “bigger = better” logic.
For a hunting rifle, that also means portability. Most of the big bore AR-10 conversions I’ve seen are pretty heavy, not ideal for carrying in the field. They kind of need the weight to counter the torque experienced from larger bores, heavy projectile mass, and large propellant mass because it swings the magazine like a pendulum around the bore axis away from the position of feed presentation. With a heavier cartridge stack due to the combined weight of multiple 200-390gr projectiles, the magazine spring really needs to be stronger as you get into those weights. The magazine needs to be designed around the cartridge to get it to reliably lift, strip, and feed up the ramp or ramps, which also need to be looked at for correct feeding at BCG velocity into the breech.
The .338 Federal actually has noteworthy performance to it that had its limelight taken by interest in more of the 6.5mms at the time it was introduced. You basically get .30-06 velocities from a short action, with projectile weights ranging from 180-225gr, no need to modify mags other than maybe increased power springs for the 200-225gr, though I don’t think it would be necessary for 5rd AR-10/SR-25 mags.
The absolute brute of the bunch appears to be the 475 Bishop Short Mag more for bragging rights than anything. The 375 Raptor and 338 Federal seem more practical.