I don't believe that one could spend too much time in the "caves" even if you wanted to. They are basically trails with rocks above one's head.
But still worthwhile. On at least two occasions I've done day hikes that travelled through both cave areas and did the high peaks trail where you go up and down steps and hang onto cables. Those are the highlights. You can easily do all that in a day (we did it, plus drive down from the Bay Area, and drive back home).
Assuming you're starting on the east side, where the camping and visitors center are located, my plan would be to park at the visitors center and go up the bear gulch caves to the reservoir, then return, either via the Rim Trail, or, via the trail alongside/above the caves trail. Then go up the Condor Gulch Trail and left on High Peaks Trail. Loop around to the Juniper Canyon trail, then down. From there, the balconies Trail to the Old Pinnacles trail gets you back. Maybe do it in the opposite direction. The flat trail around via the Balconies might seem more like a slog at the end.
Skip the Descent and Balconies bit to shorten things; come down the High Peaks Tr back to Bear Gulch in that case. This would cover those caves, thus allowing you to do see all but the Balconies in one loop. I suspect that's how the majority of people do it.
The map I linked to previously is likely what you would get in the current park brochure. Some detail is omitted, like the restroom at the top of the High Peaks area.