I find a .204 very effective to 400 yards on coyotes. The only issue I have is if there is any intervening grass or brush. I like the explosive 32 grain bullets as they don't bounce around. I do think a small frame gun is the answer and a proof barrel is a good light weight solution.
I am not a fan of any other cartridge in a small frame AR other than 5.56 or .204.
If you go with a .204 don't use the P Mags. Use the metal USGI mags and they will feed fine.
I have shot over 7000 .204 rounds with no issues in small frame AR's.