I have the Ultralite (loft 5") and have slept in the Megalite, which has the same loft as the Summerlite at 4". The Megalite has a top draft half-collar whereas the Summerlite has a full draft tube down the zipper so its probably a wash in added comfort for those features with maybe a slight edge to the Summerlite.
Essentially, you're trying to replace about 1" of loft by carrying the SL over the UL. Looking at your list of added clothing and liner, I'm fairly sure that I would be fine in the Summerlite down to 20 degrees. The question is would you? Do you sleep warm or cold? Are you generally ok in a bag with 5" loft down to 20 degrees? If the answer is yes, you'll probably do fine with the added clothing you list and the SL bag.
To examine your question much more in depth (from a standpoint of weight and functionality) we need to look at what you really gain with your proposed sleep system. I would say very little. I'm sure you'll carry the beanie and UL inner jacket regardless of which bag you use since you'll want those items around camp. As for the silk liner and down pants, you want to carry those to boost the temp rating of the SL bag down to 20 degrees. The weight added (5-6oz or so for the liner, 6.7oz for the pants- total 12 oz or so) will probably put you over the 29 oz. weight of the UL bag alone. One could argue that the down pants give you a little more flexibility around camp but at temps around freezing you're not likely to need them (remember that you're really not facing coldest temps until very early morning) so I think the effectiveness of this system is limited in temps below freezing.
By carrying the SL you will certainly gain weight savings when you expect temp lows above freezing, when you can leave the liner and down pants at home. On the other side of that coin, the UL give you more flexibility for temps below 20 degrees by adding the pants and possibly the liner.
I would look at the equation from the standpoint of hiking conditions you will be most likely to face. If nighttime lows will be at or above the freezing mark frequently during your hikes, I'd say go with the Summerlite. If you will be spending a good amount of time in conditions colder than this, the Ultralite would be my choice.
I apologize for the lengthy reply but its often best to temper your gear choices with consideration to the conditions most likely encountered and consider options from all angles. Unless, of course, you have bottomless pockets in which case you simply buy both bags!