I always viewed it as respectful ie. Yes mam and yes sir when addressing my mom and dad. I never viewed it as offensive when I was addressed as young man.
I think calling adult men and women "ma'am" and "sir" is more common in some parts of the country than in others. It's definitely Southern if you use them with your parents. And naturally they're also proper ways to address superiors in the military, or on the police force and the like.
But personally, I've never called anyone "sir" or "ma'am". Much less "young lady" or "young man".