Just swapped mine today.From KRG
Turns out it wasn't the nut portion that needed to be punched out - that just came straight out. On mine, it was the unthreaded section, originally on the RHS, where the thumbwheel recesses into, that had to be punched out.
It was pretty securely in there - took slightly more force than I was expecting.
Just noting here in case anyone else comes to try theirs and thinks something's wrong ... if so, this might be one of the rare times where "use a bigger hammer" is actually good advice.