chetos
I pretty much looked down on most non-football sports in high school. I played baseball a few years, and wrestled here and there, I tried some track but I was too slow and had to little vertical to do any of the things I wanted to do, like the high jump.
I too was at least decent at most of the sports I tried, and feel like golf's difficulty was the main thing that got me hooked. I felt like I should be able to go really low right off the bat but ended up shooting in the 120's my first few times out (I never really hit any balls or anything before I tried to play the game........it was pretty damn ugly to say the least).
Still, in my first round (June 1997) I holed a 40+ foot putt for a par on the first hole I ever played (360 yd par 4 dogleg left) and holed out a wedge from about 50 yards for a 7 on a short par five, so I knew that I had some ability.
Most of the guys I started playing with have either given up the game, or are just occasional golfers now. I can't find very many golfers my age around here than want to play "real" golf....they just want to just drive the cart around and drink a lot of beer....play the ball up etc.
Most of the guys I meet are loose with the rules.......Hell, I played with a guy a few years ago in Florida that literally took at least 20 mulligans, and then wanted my to sign his card for an "88" so he could show his friends how "well" he played! 
In the last 5 years or so most of the guys I've played with regularly are in their 60's and 70's, but they can play, especially from the senior tees! I played with a guy about a year and a half ago that shot a 70, one under par, from the back tees of a "real" course at 81 years of age! Admittedly it was a pretty short course, under 6,400 yards, but it's tricky and several times he had to hit 3 wood into the par fours, and stuck a couple of them to inside 5 feet!
I LOVE the idea that 20 or 30 years from now I could still be playing the game reasonably well!
I do play as much tournament golf in the summer as I can, usually 15-20 events per year, and get to play with some pretty good competition from time to time in them......but generally most of the time I practice alone, and early/late in the year fairly often play alone. 
Still, that's a great thing about golf......you always have the course to compete against!!!! 