Cross country skiing always means sweat, no matter how cold it is. Keeping that in mind, I would steer more towards the fleece and maybe shell if it has pit zips, or vents easily. Easiest way to quickly cool off while skiing are by removing hat or gloves. I don't know that down will help you too much in these conditions, but would be nice to throw on after skiing.
What kind of trip is this? Will you be wearing a pack?
Welcome to the nord club.