Before I came to BPL, I brought a change of clothes for a trip like you're taking, 2 nights out. I get where you're coming from, because on hot weather dayhikes, I'll bring a change of clothes and leave them in the car so I have something dry/clean to put on for the drive home.
But when I posted my first gear list on BPL, which had a change of underwear and shirt, it was quickly and rightly pointed out to me that I'd just be putting clean clothes on my filthy body, so what's the point?
I change clothes after a day hike so I'm not wearing damp, sweaty clothes in the car. Out on the trail, in temperatures you're describing, I'll just clean off in a stream, river or lake. Wash up, rinse off and then continue on hiking feeling cooler while my clothes dry out.
I do bring a spare pair of socks for sleeping,(liners, maybe 1.5 oz for the pair) because socks do get kind of grimy.
Psychologically, maybe it helps to carry a spare pair of underwear, and that only costs you 2-3 oz, but if you bring a whole new set of clothes, that's another pound you're adding to your pack.