If you had a brake so dirty you got the exit hole clogged with unburnt powder to where bullets were striking, you could obviously mess up accuracy, but that's sort of at the extreme.
Take the mystery out of it....................check out Joel on Precision Rifle network. He has a 5 minute video on how to clean your BRAKE in some lemmony dish soap type of stuff. Oh, and you're going to need a coffee cup too.