These are all very insightful comments, and things I have thought about in the past as well. I definitely agree on the teaching aspect; I'm a college professor, and though there are still many things that can occupy one's summers (preparing lectures takes a lot of time, and many professors are required to do research in one's field), in general you are in control of your summer hours, which is ideal for backpacking. You also get extended holiday and spring breaks as well (though again, this may be spent grading papers, but you can do this anywhere, so you're not having to fly on Christmas eve with everyone else). I remember one of my college professors put it well when he said "sure I'm working my tail off 7 days a week during the year, but in the summer I go fishing with my son every day." That said, I also know many teachers who hold summer jobs to make ends meet (construction is a popular one, believe it or not!).
Another solution to this modern dilemma is to move yourself to a city with nearby recreation. Not just a nice local park, or some outdoor trails, but the kind of place you'd actually seek out to go backpacking. This is hard to do, as most of these places are rural by nature and likely won't support a 6-digit income. But I have found a tremendous boost to my morale by moving to a city with nearby recreation. I grew up in Los Angeles, and while there are excellent backpacking trips in California, most involved several hours (usually 4-7 each way) of driving to the stuff I actually like to do. So for a 2-day overnighter, I would come home feeling exhausted--not from the hiking, but from all the driving. So I moved up to Portland, OR, where I can fit so much more into a weekend. One of the main reasons we moved up here is because my wife has a 9-5 profession, and we knew the only way we'd still be happy (particularly with me off in the summers) is if we could get away on a 2-day weekend with a relatively short drive. The photo you see on my avatar was taken from a hike whose trailhead is 1hr 15min from my house, and it's only 3 1/2 hours of hiking to that spot.
I also find the particular suburb I live in is filled with greenery (several of Portland's suburbs have been named "tree city USA"), and I have miles of walking and biking paths from my doorstep. Also, neither of us get on a freeway to go to work. Thus I feel much more satiated doing a weekday morning walk than I did living in a concrete metropolis, and our commutes to work feel more like a country drive (we live right on the urban growth boundary, so we drive by 5-10 acre farms on our way to work). This has done wonders to our sense of balance and ability to spend weekdays indoors.
Again, these types of places are hard to find, particularly when job searching from far away, but they do exist. My wife and I took a risk moving up here, as neither of us had jobs lined up. Our first year was spent in terror, as things weren't coming together as we had hoped. But I start up a new job as a professor in the fall, and my wife worked up a contract business as an attorney. Now, we are extremely glad we moved, and doing well with more flexibility and lower cost of living than we had in LA.
With a 6-digit income, it is likely that you have limited mobility (i.e. you move because of a job opening, not because you want to, and it has to be a somewhat large city to support you income). My first suggestion would be to get out of any debt as soon as possible (for tips, read "The Total Money Makeover" by Dave Ramsey. I typically don't like self-help books, but this one is excellent). Get 6 months living expenses saved up, and then you are ready to be riskier with a career change or a more dramatic relocation. Then, buy a house much smaller than you think you need (the book "The Millionaire Next Door" says that 7 out of 10 millionaires live in a middle-income or lower house--that's how they became millionaires!). I know several people really well who make 400-600k a year who are struggling to retire in their 60s because of their huge mortgage and affluent lifestyle. There's nothing wrong with this, but it does make me ponder the fact that there are many poor people working very hard AND many rich people working very hard. It seems like if you could get a great hourly rate and work 20-30 hours, you could still live better than most, and that would get you out of the office more than many outdoor careers. I think one of Dave Ramsey's most powerful points is that wealth has little to do with how much you earn, and much more to do with how much you save. This is how Michael Jackson can be bankrupt, while a person making $50,000 can retire a millionaire.
Anyway, back to the subject: I had seriously considered a career change to an outdoor field about 2 years ago. Instead we moved our careers to an outdoor city. Ultimately whatever you are doing as your main job will feel like work (I became an orchestral conductor to have a 'rewarding' job, and though there are certainly huge rewards in my field, the vast majority of what I do feels like work). An outdoor job may actually be worse because you will be outside working while those around you are heading up the trail recreating. Doing what you do best in your career while living in a great area can be an ideal balance.