Once you've cancelled, it usually takes about 48 hours (at least that's how long it is for me) for it to be removed completely from your orders. Etsy will send you an email to confirm that your fees have been refunded, so that means the cancellation has gone through completely. You just have to wait, and it will go away from your open orders soon...