Very interesting thread. Since I enjoyed reading all the posts so much, I'll make a contribution to it. I have been replacing synthetic insulation with down for the majority of my gear choices that are used for sleeping or in-camp or on-trail breaks.
However, for hiking insulation, I use no down, as many have said.
As far as fleece is concerned, whether I bring any depends on what I expect to encounter and how sure or unsure I am about those expectations. The higher my perceived risk of cold wet conditions, the more likely it is that I'll bring some fleece.
I will put in a plug here for merino wool (I like Icebreakers) in winter, since I can wear it for days without ever taking it off and it does not get stinky in the process.
My favorite winter system for backpacking in the Eastern woodlands that I frequent with daytime temps usually ranging from mid teens to mid 30s in general, is this: I wear a mock zip neck merino base layer (Icebreaker 240 wt). To this I add a Golite windshirt, followed by a Marmot Driclime windshirt (essentially a thin fleece with attached windshirt) with a DIY hood, and a fleece ear band, a Possumdown beanie, and merino gloves. I have a down puffy ready to take out of top of pack during breaks if I get chilly.
This gives me tons of options by shedding and adding layers as needed. The extra Golite windshirt only weighs 3 oz and lets me have just a thin wind layer over the merino, which is frequently all I need once I get warmed up and the day warms up. It also makes it very easy to get the DriClime on and off since it is slippery, and the inner fleece of the DriClime is not. By the end of the day, some sweat will have accumulated in my merino layer, at least over the back where my pack was. Once the pack is off, as I'm getting my shelter set up and gear organized, much of the moisture in the merino layer gets pushed out by body heat. By the time I hit the sack, my hiking layers are usually dry enough to be sleeping insulation if its cold enough to need them for that. More often than not, it is, especially if I am using a sleeping quilt. I would think that a fleece layer could probably replace the merino layer in this system.