It's not about whether you use designer fabric or not. It's about your unique designs. They are one offs, can't be bought in a store. I think the designs are definitely designer, I think you can definitely charge more.
Think about ways you can streamline your process. Is there a friend or family member who will help cut out pattern pieces? Attach elastic? Sew frills? Cook meals? Other things like cooking in bulk, or having the kids stay at Nans for a night, can all help you get through a busy season while keeping your sanity.
Maybe you can think about how to simplify your designs. For example, if you have a dress that has a waist sash, you could use a wide ribbon instead, to save time cutting, sewing, turning in, etc.
Best of luck!