I think a size large will fit the bill for you. I'm of a similar size and height, and love my thermawrap vest in exactly the roles you're thinking. I originally got it in size large to layer it over a size medium thermawrap jacket, and I occasionally still use them together that way.
More often now I still carry the vest but use some sort of a down jacket and layer the vest under. And of course, I often wear just the vest, and very common for me in cool-to-cold weather is to layer the vest with a windshirt (I've put the vest both under and over the windshirt, depending on particular circumstances, it works fine both ways).
Indeed pretty thin, doesn't look like it would keep you all that very warm, but I've been very happy with mine.