For context, I also own a pair of TL3 actions, run them as switch barrel setups and have barrels in 223, a variety of 6mm calibers, and 7 SAUM.... exactly what you are looking to do.
The reason a short action would be so much better is because of caliber choices available for the three TL3 bolt head options. The 223 is absolutely perfect if you want to shoot a lot, inexpensive components, easy to reload for, a ton of fun to shoot. For your middle caliber hunting gun you have 243, 6 Creed, 6.5 Creed, 7-08, 308, and many other capable hunting rounds. For the magnum bolt face you have 6.5 PRC, 6.5 SAUM, 6.5 Sherman Short, 7 SAUM, 7 Sherman Short, and even the WSM cases... all of these can send larger heavy bullets with a fast twist barrel for very long range shooting or large game. My 7 SAUM in my short action TL3 is a legitimate 1 mile capable gun. Longer barrel, 195 grain bullets at 2850 fps.
The problems with sticking with long action are that you completely lose the 223 bolt face option. No magazines, too short to feed, just not likely to work at all. In the 308 bolt face all of your long action cartridges (270, 30-06, 280/AI) are already a bit overkill for white tail. Magazine options are much larger/bulky and not sure what's out there if you wanted a 10 round mag. Certainly would work well as a hunting gun or more powerful option, but really is into the category of large game rifle. A magnum option like 300PRC is certainly a great choice for a very hot long range cartridge, but it's really overkill for most things. Cool if you want to shoot super long range or kill an elk at 1200 yards, or just one-up your buddy's gun because yours is larger.
So if I were in your shoes and wasn't able/willing to back up and start over with a short action, here's what I would do to solve your switch barrel problem.....
Choose 308 as your "plinking" round. Run a heavier contour barrel with a muzzle brake to keep recoil a bit more tolerable. Use medium weight bullets and lighter powder charges... longer barrel life, able to shoot more rounds on a range day, etc. Accurate makes a nice 5 round mag that can run 308 in a long action which might work.
Choose a nice long action cartridge as your do-all hunting round. 280AI is pretty cool and would be capable from whitetail up to elk and beyond. With a heavy bullet it's easily a 1000 yard plus gun (or farther), pick whatever contour you think would be hunting gun appropriate.
For the magnum barrel, I would suggest doing that last. After you've got the long action cartridge running you might find that you don't need/want a more powerful gun. If you did ultimately decide to build a boomer, then 300PRC or 28 Nosler or something similar is fine. Heavy barrel if you want it to be a range gun, light barrel if you want it to be a hunting rifle.