A big consideration you need to look at with RCM and its .532” case head and larger body diameter is chamber wall thickness. For a limited round count occasional hunter, it should be fine but it would do better in a chromoly steel barrel vs stainless I think.
Have a really tight-fit thread job between the extension and the tennon. It’s a 62,336psi MAP cartridge, so that helps that it isn’t a 65,000psi like most magnums.
In order to get that additional 200fps over the 338 Federal, you will generally need all the propellant and pressure you can get, though I lean on optimum propellants over chasing pressure.
The other problem is magazines. Due to the case diameter, it might not stack and feed as well.
338 Federal is the easy answer here if you must have a 338 AR-10.
Is this for hunting?