Spending summers in Denali (think days of 40s-50s, constant rain and wind, and helpin' spoonfuls of soak-you-to-the-bone bushwhacking) and winters in Western Oregon (think the same minus the schwhacking) taught me to love fleece. In those conditions, you're going to be and stay wet. If you generate enough body heat to stay warm when soaked and have the stamina to never stop moving in order to continue generating that heat, more power to you. I found that I didn't. I also found that synthetic insulation worked well when damp but sucked when soaked. So, I started packing ye olde fleece. Low and behold, it actually kept me warm when wet. And wringing it out and throwing it back on removed a significant amount of water, at least for awhile.
So, yeah, if you recreate in conditions that allow you to stay warm and dry enough that you don't need insulation while moving, pack the puffy. But for wet, cool/cold conditions, fleece is awesome. The minor increase in weight and packed volume is so very worth being able to actually enjoy trips in continuously wet weather.