I've been contemplating the same thing. The camera on the 4S is excellent and I have taken some excellent shots (including sunsets) with it. I'm not so sure I am going to keep my LX5...and my LX3 is going up for sale very soon. One of the major benefits of using a mobile device as a camera is that you can get specific functionality through software like time lapse, HDR or panorama shots. Additionally, even though it does not have optical zoom, I rarely use zoom when hiking - I prefer wide shots that show the context of where I am. If you need to get even wider, you can buy an additional lens like the Olloclip (which has macro as well), or as mentioned, use software to automatically stitch together a panorama.
There are cases where a small sensor camera does not work well: low light, photos requiring flash (camera phone flash is not that great IMO), video while moving (the best video is shot while the phone/camera is still or on a stabilizer)
Some of my favorite apps:
DMD (Dermandar) for automatically creating panoramas
Autostitch (also for panoramas but can shoot multiple rows of shots)
Luma (for shooting software-based image stabilized video)
Frameograph (stop motion and time lapse)
Camera Awesome (by Smugmug - still and video with built in image editing)