I hear what you are saying. I ask myself those questions on a daily basis. The cold hard truth is, my wife and family's needs come first.
I've been soul searching lately, trying to decide if I want to chase another dream of mine? I've always dreamed of managing several different ponds or lakes. Stocking and reducing as necessary. I make a decent paycheck and could afford to buy a farm with several ponds on it. The problem is, I would give up everything I've worked so hard to accomplish here, and I like it here.
I'm the same way about working hard. I'm either in, or out, not much in between. Mostly obsessive until I feel that I've got it mastered, then move on to something else. Gardening and investing/trading is something that I'll never master, it's a good fit.