Horses for courses, as they say.
The Sony NEX 7 and the Fuji XPro1 get excellent reviews for image quality. These are APS-size sensors, so the same size as your D80, not 24x36mm ("full frame.") I've handled the Sony and it's a nice little camera. Really little, in fact, though the lenses are rather large.
My personal feeling is that micro 4/3 is the sweet spot for the tradeoff between size/weight and quality. The sensor size is slightly smaller than APS, but that size difference allows for much smaller lenses - my Panasonic GH2 with the standard zoom is easily less than half the size of my 5D Mark II with the standard zoom. There is a large selection of Micro 4/3 prime lenses, too, and many of them are excellent.
The image quality of the smaller formats is not going to be as good as a D800e or a 5D Mark III, especially at higher ISO values. But that's the tradeoff for smaller cameras and less weight. Is the image quality adequate for your purposes? If so, great! If not, carry the heavier camera.
I would expect that the images from the GH2 would surpass what you are getting now. I've been very happy with it.
As for a full frame compact, it's possible, but the sensor size would dictate fairly large lenses, so it might not be all that small overall.