"But it is worth it and the camping at Sunrise Lakes is very nice!"
The first of the Sunrise Lakes (the most southwest) is the spot where most beginners stop, because they've just come up the big hill from Tenaya Lake, and they can't go any further. So, that is the spot where the food is protected the worst, so that is where the bears hit first. Back in the days before bear canisters, there used to be eight or ten shredded bear bag cords dangling from the various _wrong_ tree branches.
That detour is just a few miles out of the way, but there is some additional elevation gain. However, the section of the JMT that you are bypassing has few redeeming qualities, other than being slightly shorter.