As Eric and others have mentioned, best to be slightly chilled while active and have something puffy to throw on during breaks.
Some material are more hydrophobic than others, grid fleece or a loose weave sweater will have lots of dead air pockets that won't hold water. They will feel warm under a windshell even when wet.
As a comparison, I have a dense poly fleece mid layer that is about the same thickness as a grid fleece that I normally wear.
The grid fleece isn't as warm worn alone, but is almost as warm as the dence fleece under a shell.
The grid fleece actually seems warmer compared to the dense fleece when wet. It's also lighter and dries much faster. I have even layered two grid fleece sweaters in really cold wet weather, but it was too hot while active.
Still, it's nice to have that puffy down jacket to pull on in camp or while sleeping. Just don't hike in it:-)