Relative humidity will change the location of the dewpoint - and at a low enough RH and a warm enough temperature, that can mean the dewpoint is outside the bag rather than in the layer of insulation. That can make a substantial difference in how much moisture the insulation absorbs.
But to get back to the OP's question, the big issue with having the lining of your bag be the VBL is that you can't adjust the insulation without venting the VBL, which is a major flaw. To use a VBL effectively you must maintain the temperature and humidity within a relatively narrow range: too warm, you'll sweat. And if you sweat, and then vent to dry out, you dump a lot of heat and start a cycle of too warm and too cold combined with a lot of heat loss. So you want to be able to adjust the amount of insulation without venting the VBL. This works best with VBL clothing, as you can even add layers of clothing if you want. With a VBL bag liner, you can unzip or otherwise open you bag/quilt to adjust the amount of insulation with only a minor opening of the VBL. But if the lining of the bag is the VBL, you have no way to adjust the amount of insulation without venting, since all you can do is open up the bag and thus vent. So it's not a good way to go.
Now if you are using a VBL liner or clothing, you could also have a sleeping bag made entirely from non-breathable fabric, since it won't accumulate moisture and thus doesn't need to dry out. You do need to have part of the shell breathable so that air can get out when you pack it and back in so it can loft up.