Now I don't know how to get the order out of my open orders folder. There is a blue "cancel transaction" link, but I"m not sure if this cancels the order or cancels the refund.
It will cancel the order, not the refund. If you've already refunded then you can go ahead and cancel. It will stay in your "open orders" folder for about two days until Etsy processes it.