Instead of thinking that you are a shy person (or whatever trait they are judging), they might think you think you’re better than them or whatever.

I remember in high school overhearing a classmate say something about a classmate who happened to not be present that day. It was something along the lines of

“he doesn’t really talk much though, he seems stuck up”

Hearing that honestly rocked me, I’d never even considered that line of reasoning before. He wasn’t talking about me, but I was pretty shy too (well, socially anxious) - I’m almost certain he thought the same about me, or even said it aloud when I wasn’t around. I immediately thought “Does everyone else see me the same way? Do I appear stuck up and unapproachable?” Anyway, I haven’t seen that dude in years, don’t even remember his name, but I never forgot that line.