I haven't noticed a problem with next to skin feel as I usually sleep with a base layer, but even without, I don't think it's so much the "feel" as the sweat/condensation (which in turn feels moist and not great). i.e. you can solve both those problems with adequate venting, which with a quilt is easy in theory. One problem is that if you are a deep sleeper or your body doesn't have the auto-reaction memorized yet to stick a foot/arm out when warm, then you might not wake up to vent (or subconsciously vent) until you're drenched in sweat. This is where the "feel" comes in to play it seems...if you're not venting you're going to sweat and it's going to stay inside the VB regardless, having a base layer on may make that situation less uncomfortable, but then again it might also delay your body's tendency to wake/vent and actually make the problem worse. Did that make any sense?
At any rate, I sleep with base layers and just let the quilt setup do the auto-venting. Depending on the temps I'll start either at: with it half on, mostly on with a limb or two out, completely draped but loose, wrapped all the way and strapped behind me, or fully wrapped and with the neck cord tightened. As I get cold during the night I'll just progressively tighten it up just like pulling more covers on at home, (and adding hats and/or other layers if necessary, though insulation layers should be draped over the quilt rather than worn under it due to the VB). I haven't had any sweat/condensation issues, and I've been happy with the setup so far.
This isn't technically using the quilt as a full vapor barrier until you're at the fully wrapped phase, since a partially open quilt isn't acting as much of a vapor barrier. However, I think this is actually a benefit to using it with a quilt. I've used the same quilt at 15 degrees as at 50 degrees, although I don't know how much of that is helped by having the silnylon layer. I might have been just as comfortable at 15 (not very) in a comparable quilt made of breathable material.
I must have been typing all that when Roger was posting, but his comment sums it up nicely...keep it cool and your body won't overreact.
And as for the "hardliners," read Jack Stephenson's take on VB as the very next-to-skin layer for a strongly opinionated VB proponent's views! www.warmlite.com