No, this woman don't understand how important Ellie is to him. As far as Joel is concerned, she's got no fucking clue.

"Yeah? Well, I don't know him, just like I don't know you, and I could give two shits how important this Booker guy is to you."

Even if this guy hasn't done anything to hurt Ellie, Joel is going to make sure this guy gets the message loud and fucking clear to stay away from her. He's got reason to trust this woman, and he sure as hell has got no reason to trust this Booker asshole.

Joel steps aside to let the woman keep walking, and says, "Get movin'."