FMJ "can" work, and it can work well...it just doesn't work as consistently as a bullet designed for killing game.
Ive killed coyotes with the Mk 318 Mod 0, Speer Gold Dot, 69gr SMK, and XM193 in 5.56/.223 out here. The SMK and XM193 usually have to hit the shoulder to be as impressive in dropping a dog as the Mk 318.
When I retired and moved out to the ranch we had a huge coyote problem. I killed 26 in the first year, and a total of 48 over the first two years just on our 700 acres. The numbers are much more acceptable now, but I'm still shooting one a month or so.
100-200 yards any soft point in the 55-64 gr range is going to be good...you just have to find ammo that your gun likes. Varmint bullets are great too...you may not be able to get quite as cute with your angles, but well placed shots offer impressive results.
Haven't ever shot a dog with a .224 diameter TSX, but I've actually had great results with that bullet in 6.5 Grendel and 6.8 SPC. The SST is devastating on shoulder or lengthwise shots, but surprisingly not nearly as effective on traditional hunting shot placement. Ive actually had just as many coyotes run off after a double lung with the 6.8/SST as I have the .223/SMK. I find them later, but they go 100-200 yards.