Jack is apologizing to him. Jack is... apologizing to him? And not for the fight or bringing him out here. It takes Oliver a few seconds to process exactly what Jack is sorry for, and then he goes over the rest of it, searching for what made the spirit think he was responsible.

"What I did wasn't your..." Except he doesn't think that's exactly it. He can't tell Jack that he wouldn't have killed the kedan either, because truth told he might have. And it would have been excessive. Logically that isn't hard to parse. "Why are you sorry?"