There is one situation where you would NOT want to wear even dry clothing to sleep.
If your bulky clothing puts pressure on the inside of your sleeping bag and collapses the down, you've just created little cold pockets and you will feel that cold seeping into your body. I have this problem with most sleeping bags and couldn't hope to wear a bulky jacket inside my bag. It would be very counterproductive since I would collapse the down in BOTH the bag and jacket. In a bag, I have to drape the jacket over top of the bag. This problem is solved with my quilt which offers sufficient room for me to wear the jacket.