It kind of depends on your day clothes. If they are wet, you probably don't want them in your sleeping bag. If you spilled your dinner on them, and you're in bear country, you probably don't want them anywhere near your tent, much less your sleeping bag. If you happened not to look before sitting down when in cattle country....
You do have to wash your sleeping bag more often if you wear your grubby day clothes inside.
I always used to sleep in my day clothes. However, since I carry a base layer for insulation anyway, I've taken to sleeping in that. If I wear them outside (which I do when it's cold), my day clothes are over them, so the base layer stays relatively clean. Although I'd never bother taking separate clothes just for sleeping, I do take a pair of 200-wt. fleece sleeping socks. My feet love them!
There have been a few times when I've been so tired that I couldn't be bothered with changing clothes at bedtime and just crawled into the sleeping bag, grubby clothes and all!