I've been transitioning to a minimal shoe for the past year or so, and I'm a not totally clear what the limit is for me either. Last year, I used an NB 101 on trail runs and backpacking trips, and it was fine, although not all that minimal IMO.
This year, I started with a merrell trail glove on day hikes/runs. I love the feel of hiking with these especially on rocky/technical terrain. You have to choose your footing more carefully, which I think adds to the fun. I did one 3 day trip a few weeks ago on the colorado trail, with three 27 + miles a day. My feet and muscles felt great at the end of each day. I'd say fresher than normal, I suspect from a softer/more efficient stride.
However, the following week I went for another 3 day trip on the CT, this time with 30 + miles each day, and by the end of the 2nd day my feet were bruised and I was miserable. I bailed. I attribute the bruising partly to 20 miles of fast hiking/jogging on the first night done with a headlamp over rocky terrain. I've since reconsidered footwear, but more so my itinerary for weekend trips ;)
From what I can tell, if you are attentive about foot-placement over a long day of hiking, you can get away with a very little shoe. For me, I think the trail gloves are perfect for hikes less than 20 miles a day, and I'm still experimenting with mileage beyond that. I think the most challenging terrain for minimal shoes is little bits of loose/sharp shale. Hard to pick your way through that without slowing down quite a bit.
I'm weighing taking the merrells on a two month trip to new zealand this nov/dec. At this point, they are my preferred mountain footwear, but I'd hate to have to deal with bruised feet on a long trip like that.
A lot of people seem to love the innovates on these forums, but IMO the lugs on even their most minimal shoes make them pretty lousy for rocky trails. I think these were specifically designed for fell running over muddy UK terrain. Having soft rubber lugs like that leaves your feet more vulnerable to pronation, like more built up shoes, and can make for some sloppy footing on rocky terrain. Just my experience/opinion, I'm sure they work well for others, and maybe they have some less lugged shoes in their lineup that I haven't seen.
Sorry for the ramble....in summary: toughen your feet as much as possible and make sure whatever you take is well tested on the type of terrain/mileage you expect to cover. Hope that helps!