"While you are going uphill, you tend to generate a lot of sweat. If you have a cotton t-shirt, then the sweat will hang in the fabric. Now you are generating heat that is going into damp fabric, which feels warmer. Once you get up over the pass and start down the other side, you will not be generating much sweat. The cotton shirt with sweat included will get cold since the sweat is now evaporating. So, you get chilled on the way down."
As a long time desert rat, I can completely verify what you are saying... It's especially true when cycling, as your downhill speeds tend to be a hot higher than when hiking.
Now, that being said, I spent many hot summer in cotton long sleeve shirt. I used to weld for a living, and when synthetics catch a spark, they turn to napalm, where cotton only smolders and is easily put out. But the kind of cotton makes a difference... I would stay away from knit fabrics like your normal cotton T's. They get wet and sticky, cling to your body and create a high humidity micro climate, and feels like it's slowly suffocating you. Woven fabrics like dress shirts are much, much, better.
Also I would have to say for personal experience, that while a cotton T is passable, I would strongly recommend AGAINST cotton pants and underwear unless you want some serious discomfort. Once the heavy stitching is the crotch, groin, waistband areas get wet, they stay wet, and abrade you skin to no end.