As a buyer, to be honest, if I were to -request- a custom order, I would expect to have to pay at least half of the fee, if not all of it up front. That might just be me. I don't know. I don't like to waste people's time by requesting something. Plus, if you don't pay them upfront, the seller might keep putting it off or not do it at all.
As a seller, I would say request -at least- 50% up front. That way, if the custom order actually turned out to take less time and materials than you had originally thought, you can lower the final payment appropriately. Or if you're having a sale, you could be generous and alter the final payment so that it fits with it,