Isn't that exactly the point I made? You mentioned that people who rant about people being late annoy you. You don't have the power to change them, but you can adjust your own action and be on time.
I am not saying you should btw, I am just saying your argument could be interpreted in this way as well.