There is also the complimentary form when addressing the elderly.
I always viewed it as respectful ie. Yes mam and yes sir when addressing my mom and dad. I never viewed it as offensive when I was addressed as young man. And years ago I remember receiving mail that was addressed to Master.