I second Sword Lake, great for swimming and fairly easy to reach. I have not taken the kids there yet but I think it's doable (depending on age stamina). This time of year you'll likely have some people around. The road in is a little long, rough, and dusty but won't require 4WD.
Another to consider is Devil's Lake near Bear River Reservoir. The hike is only 1.5 miles, lake is great for swimming and camping. I've been there twice this year. Once with my little boy, once with a friend. There will probably be another group or two sharing the lake. The road in gets pretty rough and requires high clearance, 4WD helpful. This keeps some people away, most of the people in the area are riding off-road vehicles on/near the road and you never see or hear them up at the lake.
I've done a few hikes over the past few weeks in the 5-6000 ft. range and the mosquitoes have been out in force every time.