"I just don't like dealing with Sierra storms in the snow"
Very few like snow storms. On the other hand, the storms will chase away all those who are faint-of-heart.
The risk is getting up to Badger Pass. Highway 41 is OK, but then you have to turn east on the Glacier Point Road for those 4 or 5 miles to Badger. If you are lucky and there hasn't been any snow for days, then you might be able to slide up there with an R1 restriction on the road. However, if the road gets just a little worse, they will up it to an R3 restriction (snow chains required, all vehicles, no exceptions). Of course, that makes the 4WD Jeep drivers mad.
The reason is that this short section of road is not banked right for winter driving at any speed, so cars (typically descending) slide out of their lane into the opposite lane, and that gets bad results.
The last time that I was parked at Badger, it took me an hour to shovel my car out. Then I had to drive (with chains) down to Highway 41, and then had to go with chains about halfway down to Yosemite Valley before it all turned to rain.
The smart thing to do is to load as many people as you can into a 4WD vehicle (with chains available). That way, the entire team can deal with the vehicle at once.
You have been warned!