I don't really get any challengers, I'm a pretty big guy to just go and fuck with (not fat). I have in the past, but I don't put myself out there to be in that situation. Like going to a club. It's no interest of mine and I'd kick the shit out of some guy for grabbing on my fiancee. So, it's obviously best to stay away from something like that.
Choice of words works best. I've been really close to getting into several fights over retarded things. I explain to them that it's stupid and why, but if they want it, come get it. Most typically realize, yes, it is quite moronic to fight over something stupid, and intelligently walk away. Though some were not as smart. Others are there just to intimidate. They'll talk all the shit they want. They may even get up in your face to see if you're man enough. Unfortunately for them, they find out that I don't talk much and I will gladly hit you for invading my personal bubble.
If it comes down to it, I don't back down. I'll tell you to fuck off, and show you that I don't take you seriously by walking away. I don't mind a little confrontation though if you're adamant. I'll definitely kick the shit out of you, if you really want it.
BTW, all that "let them swing first" bullshit, is absolutely retarded. That's how many people get caught off guard and end up in the hospital. Wait for them to "pose" a threat to you (legal purposes) and then tee off on their face. Never underestimate your opponent. Chances are, that's what they are doing to you. Utilize it and break them in half.