I would personally not stop offering PayPal. As a sometimes buyer, I like having the option of either or, especially as I sometimes don't have money on my bank card and have to pay with PayPal unless I want to wait a few days for the money to be transferred. And when I'm buying, I don't want to wait. As a seller, I would hate to miss out on a sale because PayPal was not available. Plus, I still get about half of my sales with PayPal :)
But in the end, it's your shop and it's up to you. If you think it does more harm than good, then go ahead. It's probably the right decision.