Flyers are just as much as part of a group as any other. If you pull one and call it, that's not a flyer, but flyers count.
I would bet that it is in fact a combo of your barrel not quite being where you want it to be, the load not being as good as it can be, and you not shooting as good as you will be able to with more practice. Of course the load is something that you can adjust easily and you get to practice while you do it. There isn't a lot of concern generally given to shooting out any 223 barrel, and it's one of the cheapest to load and shoot, so practice and develope that loaf. Just be careful loading. I would highly recommend finding someone to learn from even for a few sessions. It will help you more than you realize. I would be you can even find someone here that is within an hour or two away from you and would be happy to load with you for an afternoon. It is so valuable to go through it all and pick up little tips and habits that #1- keep it safe, and #2- will save you a lot of heart ache from doing stupid stuff (as we've all done). There's a thread about our mishaps on here but I can't recall the name.