I did that and it works well (using 5 mm silk). What I liked in addition to the weight and price is the possibility to dye to the color I want (dye raw fabrics and use more than you will need, the silk shrinks a bit).
The major drawback is probably the durability, I won't use silk for item I plan to use frequently, but this was not the case.
Another thing to consider is your sleep mode. The silk is not slippery enough to turn easily in the quilt instead of turning with the quilt. For me, turning from one side to the other requires to be at least partially awake and hold the quilt by hand.
Regarding the temp rating, there is certainly some loss compared to less breathable materials as mentioned above, but I don't think it is large difference. I've used the quilt down to temperatures usually listed for all nylon equivalent.