I think also the more you spend time outside, pushing your own limits, the more you acclimate and adjust. Someone like Skurka seems to spend A LOT of time out in the wilderness--maybe not consistently, all the time, but long stretches at a time.
If you think Skurka's tolerance for cold, etc. is amazing, well he doesn't hold a candle to someone like Wim Hoff, or the average long term Tibetan Monk that still lives in Tibet. These folks might blow your mind with what they can tolerate cold temp extreme wise.
I remember seeing a story on the weather channel a some years back, wherein the media were looking at a guy who lives in a more northern and colder part of Alaska. Some years back, a friend of his bet him he couldn't go all winter without wearing his coat. Well the guy won the bet, and decided to stop wearing a coat altogether and has been doing this for many years since. Seems to be, not just purely a physical thing, but also maybe a mental process as well.
There is also the well known story of the wild boy of France. A youngish boy was found, who had apparently been living in the wilds of France by himself for a long time, and became completely "wild". Didn't speak the French language, acted very uninhabited at times, etc. What i thought was interesting about the story is that the boy, even after they tried to "civilize" him, had a fondness for frolicking in the snow in the buff, and seemingly perfectly ok with the very cold temps. It's been awhile since i read about this, so may not be completely accurate on all details.
I think we humans can learn (or maybe it's more "unlearn" certain limiting habits of mind and body) to adapt to much more than we do currently.
Interesting topic anyways, so thanks for bringing it up.