People who annoy us are a mirror. The fact that they annoy us indicates that they have touched a nerve. If they hadn’t we wouldn’t even notice them. The nerve they have touched is something sensitive in us. Something that causes discomfort.
When we think “They shouldn’t have done that”, “They shouldn’t have said that”, “They shouldn’t be like that”, the “shouldn’t” is because they have transgressed against our sense of should. What we should say, what we should do, what we should be.
The trouble is that we know what we should be and that we’re not. This hurts and we want it to go away.