It doesn' make any difference if you are a buyer or seller. All NEW messages must be answered or put into spam.
You answered your own question....
"I talked to a lot of shops as a buyer last month and didn’t respond to them all right away,..."
Etsy is pretty clear about how the message response rate is calculated in help center here https://help.etsy.com/hc/en-us/articles/4403058372503-How-to-Become-a-Star-Seller?segment=selling
You have to reply to ALL initial messages. The system is completely automated and doesn't know if the message is from a buyer or another seller. It isn't reading every message and making a decision if it's from a seller or a customer or what it's about. It's a computer program that calculates data. It just sees a message and calculated the response time. That's just how it works
If the star is important to you then it may be a good idea to read or review all the info Etsy provides about the star seller program.
Bottom line...you have to either reply to or 'mark as spam' ALL messages.
It takes just a few seconds to mark spam as spam or send a quick thank you reply for nonspam messages. Or you can set up an auto-reply every 5 days.
If you want your seller and buyer account to be separate then you will need to create a new account just for buying.
@HollarityArt: "Why was my score lowered?" Because the Star Seller metric for messages is very simplistic and binary - either you replied to (or marked as SPAM) that first message of EVERY new incoming thread within 24 hours or you did not.
As it says right on your Star Seller page and elsewhere, you need to reply to the FIRST message of EVERY NEW INCOMING THREAD within 24 hours. It does not matter who sent it, the subject matter, if it is a duplicate or blank or anything else. If it is SPAM then it should be marked as such. Remember that the Star Seller metric is NOT all new messages but only the first message of all new incoming threads.
A controversial workaround : if the unanswered new thread is not a Help Request, at least for the time being, it appears that marking an unanswered new thread as SPAM (even after the 24 hour deadline has passed) will cause your reply percentage to change in a day or two if the original message was within the current star seller period (but not retroactively). Note that subsequent messages from that account may go directly to the SPAM folder.
You may want to reread the Star Seller rules so you are not surprised by something else.