I've removed it by putting a drift to the bottom edge of the bearing, not the top.
But that damage is sanded out and greased up and put the new one on. Make sure you get the dust seal under it first and put a pipe of the right side - and use a hammer not a hammer drill.

In fact if you had a press, the cleanest way to take out that bearing is to press the stem out. Anyway its not too crazy so its gonna be fine.

Dremel to cut a groove is also a good idea. I just rarely have had to do it to a pressed in stem instead of a welded stem, so I didn't think this needed it. Ofcourse I had access to a press so I could afford to dream up the solution.

Cool.
Srinath.