Not true. I wear mine all the time when I walk out the door of my house. I got use to it after all the gas station robberies I've done.
Actually, If I'm going from say the hardware store to the gas station a few blocks away, I don't even remove it. And just wear it into the house so that way I don't forget it in my car. I have my high tech 95n (or is that n95) designer mask in my home office and a bunch of hospital quality hanging in my car. It's just no big deal.