I am constantly deleting people from my circle. Never had any complaint.
My circle is full, so to add someone, I have to remove someone else.
Not only is my circle full, but I have been added to a lot (!) of other circles too.
Even though my favorites are set to private.
Just like you I can hardly get my activity feed to load. It is very slow.
Circles, as apposed to favorites, are more of a social tool, not a commercial one. So I would not worry about shop views, etc., when removing someone from your circle.