i saw at least one person who does the same as I-- that is, wear my wind shirt/ pants if necessary.
if i can, i wear only my hiking boxers but if that is too cold, then i don the windshirt and pants, which creates a bit of a vapor barrier and warms me up.
the only downside to that which i have noticed is that you get a bit moist inside the wind clothing b/c the breathability seems to be impacted when you are inside a sleeping bag.
another alternative i use is to just wear whatever insulating later you have-- for me, depending on the temp and season, i use a WM down vest (late spring, summer, early fall) or a Montbell UL inner (remaining parts of the year, sierra high country 10k+, cold snaps) at night to sleep in with no shirt on underneath w/ the boxers.
i thought those would get nasty quickly but they dont, surprisingly. i wash them about every year or so with the appropriate soap and they are solid. same goes for WM bags. but i do think that your gear probably does not last quite as long this way.... for me, i dont worry about that b/c then i have an excuse to get something new/ lighter! those items though, over the last 3-4 years i've used them-- logging 200-300 miles a year-- i havent had to replace them from use/ cleaning.
in the end, really, its hard to justify that extra set of sleeping clothes if you want to get the weight down. but like i said, we all have to indulge from time to time. also, its dependent on the type of backpacking-- if you are going to be hanging out at camp for any length than it is nice to change after a day of hiking and hang-out in fresh clothes. for me, who mostly solos or hikes with another like-minded partner, its set up camp, sleep, wake, and move on. no need for extra clothes then.