With the male friends that I know very well, I use the phrase "Hey Boy" when I first talk to them. Another way of saying "Hi (and their first name)". They are very comfortable with this.
I think calling adult men and women "ma'am" and "sir" is more common in some parts of the country than in others. It's definitely Southern if you use them with your parents. And naturally they're also proper ways to address superiors in the military, or on the police force and the like.
But personally, I've never called anyone "sir" or "ma'am". Much less "young lady" or "young man".