An old lady decided to crush my bike and break my arm with her car, leading to a radial head fracture. I didn't need surgery or even a cast so your situation obviously seems a lot worse, however my advice would be to start some form of physio right away. Within even a few days of not using your arm you will start to feel the muscles begin to tighten, a short period of non use as it heals can lead to months of recovery work.
My personal opinion (no I'm not a health worker) is because you've got a fracture, let that be your guide for physio. Start off really lightly without any sort of pain killers and move your arm in different motions until pain tells you the limit (there will be aching and pain but you'll know what I mean if you try to push to far.). The idea here is that you should not be able to cause further damage to the fracture without the pain telling you that you are stressing it too much. Good luck