> I just need the Goldilocks of fleece vests :)
Indeed. As the temperature drops, activity level significantly changes the absolute amount of insulation. The colder it gets, the more you want different insulation for when you are active, when you are are at rest, and when you sleep. Hence the common light to mid weight fleece (or insulating softshell) for activity, belay jacket (or some other high loft garment) for at rest, and a sleeping bag or quilt for sleeping.
One way to to make a vest do all is aim to vary activity level to stay warm when you are active, and skip "at rest" and go stray to your sleeping insulation which you vent until you do sleep. There are some thru-hikers I have met that do this. No insulation other than there bag. I am not comfortable doing this, but for someone who is trying to go as light as possible, this can work. You just want to make sure you camp just before a big up to help you kick start warming up in the morning.
The other approach would be learn from folks who are very into mindfulness like Buddhist monks who use fairly minimalist clothing in a wide range of conditions.