If you used a reasonable durability cuben fiber on the exterior (1oz) and .33oz cuben on the interior, no groundsheet, no bivy, no shelter, just a torso pad with the pack under the legs, the weight savings should be evident, not tremendous but I am comparing this to using a w/p bivy that can weigh 14.5 oz if you use the MLD soul bivy with the wire brim. So depending on the weight of the quilt you would be saving 14.5 oz + 1.7ish oz for groundsheet-added weight of 1oz cuben fiber exterior+saved weight from using .33oz cuben fiber interior-weight of valve.
However this is mostly about simplicity and this appeals to that. Plus messing around with your gear is half the fun ;)
Just imagine just taking off your pack, sliding out your pad and unstuffing your quilt and being done! No worrying about getting your quilt wet either! However I am already quite pleased with my bivy sac currently so I will most likely stick with that since this would be an expensive switch to get this custom made or attempt to make it myself.
Crazy ideas aren't always bad ideas. Just usually