We've done a three day two night group car camping trip with a 2 or 3mo old (I don't recall exactly- he was born in July and we camped in the fall) where the temps got down to the low 30's. We've always had our babies in our bed with us so it was natural for my wife to have the baby in her bag with her as well. Nursing was difficult though, for her to be that exposed (both her and the baby were clothed). I told her they need to make nursing shirts so she could whip it out to feed the baby w/o losing all the layers of warm clothing, and later we found out that they already make stuff like that.
The unexpected problem we had was with the older kids scooting out of their bags during the night. The older boys slept in the all boys tent with a couple of adult chaperones, and I also had my 2 and 4 yr olds in my tent with me and my wife. We all had kids occasionally scoot out of their bags, to be found curled up and shivering on the bare tent floor. I ended up unzipping my bag and using it like a blanked for me and my 2yr old since he wasn't going to make it through the night in his bag.
So, as long as you have a good wide bag with lots of girth, I'd say just go for it and take the baby in the bag with mommy... Wear layers, because when shuffling around to switch sides to nurse on during the nights (assuming you guys nurse) will suck in a lot of cold air.
edit- he was 5mo old, turns out we did the trip in Dec. found some pics with dates... And the days were warm- t-shirt warm, so it's not like the baby was freezing 24hrs a day... Dunno what daytime temps you expect, but it's less stress if you know you only need to worry about keeping the baby warm through the nights, rather than the whole day too...