Although I do most of my backpacking in Yosemite, I think the Labor Day Weekend would be a good time to stay away from Yosemite. The late spring and early summer season was kind of messed up due to the snowpack, so lots of backpackers delayed their trips until late season, which is now upon us.
On the other hand, there are always places to go in Yosemite that are seldom visited. In general, those places are in the far north end of the park (away from the PCT), and also the far southeastern end of the park, for example east of the High Trail near Isberg Pass.
You just have to decide your goal. Do you intend to simply pound out lots of trail miles, or find wilderness solitude, or find wildlife, or climb a peak, or something else?