I have been doing that also, but have ruined several SP wheels, who are so thing at the hub, that the outer part of the wheel pulls away, leaving the hub and axle in the body.
I have decided to use set screews from now on and be sure the axle hole in the body is loose enough to remove easy as John does above.
I've been using the braided fishing line method suggested by cooleyracing in his post dated 6/27/12 in the thread "Stuck Axles" under Wheels and Axles and haven't looked back! Works like a charm.