I only really like them for active use, so unless you're going to use a VBL, a down vest might not be the best idea.
For active use, i've found that with some kind of long sleeve shirt underneath (and like Justin B., i like hoodies in combo), as long as my core is warm via the vest, i can handle some relatively cold temps. Yes, my arms might get a little cold, but not too uncomfortable.
Otherwise, i would sometimes use a down vest for inactive use, as my main outer insulation is a Stoic Down Hadron jacket, which is not super warm by any means (though quite efficient at using what down it does have), and so i may throw over my down vest to add to warmth. Similar principle as the above active scenario in that sense, but not quite as pronounced. Enhanced circulation via fast movement or exertion really helps in that regard. But still, really warm core, seems to help warm the extremities also.
I'm a fan of light weight, but even more a fan of adaptability and flexibility in my systems--getting the most out of the least. Vests have their place in that, i think, but for me more as an add on rather than bare bones.
What i would think would be pretty nice is a good quality grid fleece vest with a hood.