Yes, I was gonna say just that: how about "I don't want to be mean, but sometimes it's necessary". I don't want to be mean, but I'm gonna be anyway because Very Good Reasons.

Also something to think about, not sure about other people but for me "I’ll stop now" is the worst. Because one will always come up with terrible meaningful things left unsaid, even if the original meaning was mild at best. "I want to say something but I won't" is ominous.