I fear offending people, for two reasons: first, I empathize strongly with the feeling of rejection, and I don't want to cause it in others; second, I fear that if I offend someone, and get on their bad side, then they'll try to make my life difficult, and I don't want someone going around trying to make my life difficult.
In both cases, I think I exaggerate the consequences. I suspect that I fear other people will feel a stronger sense of rejection or offense than they actually will, and I suspect that people are more likely to put me on their "bad" list than they actually are.
I tend to think about this issue as binary, rather than a gradient. It is very important for me to make sure that in everyone's mind, I have not wronged them, or have apologized appropriately if I have.
I'm also pretty unpracticed in the art of "breaking up" or "rejecting" someone, and I fear that if I gave it a try, I would fail pretty catastrophically. Usually I'm fine failing a lot to get something right, but when it involves other people, I'm not fine with it, so I don't know how to learn the skill.