Joey, There is a possibility that I may be hiking in the Smokies/high elevation NC late in May/early June. I have quilts in much the same range as yours (20 degree and one summer one, both MYOG).
If I go I will be taking my summer quilt (10.8 oz total weight) without hesitation, but likely take my down puller over and fleece cap also. With all of these items I can push the quilt down to upper 30's. If you are at high elevations (5000-6000 feet) you can certainly see temps in the 30's in late May, possibly even sub freezing. If you are sleeping at lower elevations than the weather forecast in the 50's is much more reasonable. I've found the temps can be 20 degrees lower than forecast at the upper elevations.