He can see Damian holding back but, of course, couldn't even guess what it is. But Damian's explanation makes a little more sense. Maybe this Jason isn't as attached to reality as he should be. But the next statement causes Dick's anger to flare.

"How fast does he replace us?"

All of this is making him feel rather disposable, on top of being irritated on the other Robins' behalf. But it's clear that his anger isn't directed at Damian.