It's simple. As people, we tend to notice the annoying things or people more than the multitude of peaceful and unintrusive. We notice who addresses us more than those who ignore us.
A few yeas ago, a Palestinian student of mine said, "Black people really like big rims on cars." To which I said, "You think most black people in the US really like big rims?"
He responded confidently, "Yes."
So I educated him, "You are basing this on the few people you pass that take a moment to comment on your car. You are totally oblivious that the majority of people do not care one iota about your car."
I continued, "This is a stereotype that was fueled by your limited perspective."
My point being to everyone who has chosen animosity toward you is that... It's always "not all" because truly "not all men" do this. Not even most.
Most men live exactly as explained by this "middle aged bloke" (as you described yourself.
The reality is ladies, we are more likely to notice the loud obnoxious or rude and intrusive men than the many who completely went on living their lives without giving us a fleeting thought.