I'm with you Nick, I think the coldest I've experienced is 119F ;)
Anna, you're picking up on what I've noticed too. That said...
When I lived in Boston I got caught in the 2005 blizzard crossing the Charles River. I had about 2 feet of snow, temps right around 0, sustained +30mph winds. My gear was skate shoes, hoody, leather jacket, jeans, no gloves (I'm from the desert after all). Took me 20 minutes in an ice cold shower to warm up my extremities. Even though icy, the water felt like it was boiling my feet and hands.
Since then I've come back to my beloved 6 months of summer.