Yeah, Four didn't really see the funny side, did he? And I don't really blame him - crispy!Master was definitely the least likeable of the Master's various incarnations, I'd say (yes, even including Roberts!). But yeah, the fact that the Doctor is too controlled and compassionate to really give the Master that reaction he wants - the one that'll prove to him that he's really got under the Doctor's skin - that drives him mad, I think.