First of all, what do you call a mid-layer? I think of something like a "heavy base layer" like Patagonia R1 or Cap4 or Power Stretch, on up to 200W fleece. If you are talking camp/belay use with minimal activity, then I think of puffy layers on the order of Patagonia Micro Puff, First Ascent Igniter and on up to lofty down stuff.
If you are thinking of a thin insulated jacket, I think the lighter insulated jackets like the Mont Bell Thermawrap or Patagonia Nano Puff don't deliver much: you get a wind shell and a light insulation layer, which a windshirt with hooded R1 or Power Stretch layer would meet or exceed--- with more versatility. I would use one of those hoodies with a puffy vest for the next step colder and that is my recommendation for you to try with your ski shell. I prefer synthetic, but for camp use and no rain, down can be very light, compact to carry and warm. A vest is just nice to have on day hikes and cool summer mornings when you want just a bit more over your base layer until you get moving.
For zip-off pants on the light side, look at the Outdoor Research Equinox Convert pants. They are light without being too fragile and they are free of cargo pockets-- just plain front/rear pockets, so less on the Inspector Gadget side of things. Ex Officio Amphi Pants are good, and the REI Sahara pants are excellent in hot weather, but much more on the baggy side. I'm not up to current models on Mountain Hardwear's pants, but their track record has been good for me.