Keep in mind the stretch of road from Lone Pine to Whitney Portal. It is only 14 or 15 miles, but it is very hilly. Cars can't go at highway speeds. It might take the greater part of an hour to travel, depending on what vehicle is in front of you. Also, on many summer days, the overnight parking lots at Whitney Portal will be jammed, so often drivers have to cruise around looking for a slot.
Just to make it more complicated, when you park a car at Whitney Portal, you can't leave food in your car. There are bear lockers for food storage scattered around the various lots, but that is just something else to soak up time.