The Doctor -is sometimes reminded of his professors at the academy when she gives him that 'stop jerking me around' look. he's not trying to annoy her; he's only commenting on it (for the same reason that he always calls in to her radio shows when the Master does, he'd thought he'd reassure) because he doesn't think the Master should get away with it so easily- pointed a finger at her.

"See that right there? That 'yes yes, you silly man, but do you have something to say that is actually worth my time' thing that you always do to me? You should try that attitude with him. I bet he'd not be quite so entertained by that reaction."

He's not entirely sure why she seems constantly annoyed by him. Maybe it's just the Master, or maybe it's because of his and Crowley's tiff. Or maybe it's simply that -if Rose were here she'd tell him he was being rude, and Una would be bemused and ask a million clarifying questions, and John would just think the goat was funny- he doesn't actually make any sense most of the time.

But Cayce confuses him.