I am also in this seemingly exclusive club of no poles.
I agree with the simplicity aspect that Don points out.
In thick scrub poles are a plain nuisance and on rocky terrain I have a hard enough time concentrating on where to put my feet let alone poles aswell-that would slow me down considerably and probably make things harder-on small boulders momentum is often key.
Poles increase the oxygen demands of the body as more muscles are engaged-this means more calories are needed and overall you will be more fatigued at the end of the day. Thats probably the main factor against my use of poles for long distance and heavy hikes. I have dodgy ankles (one has a tendon conveniently ripped off which allows it to roll very easily), and I find that it is when I am not concentrating on foot placement that I roll them-every single time! If I have to concentrate on poles as well I feel that I am more likely to roll. It would take remarkable upperbody strength and balance for me to support a heavy pack and my weight on two poles to stop an ankle rolling. I can do 10 full body dips (a waste, dont need to) but there is no way I can support myself on two poles alone, let alone lift myself up, and with a pack.
I have too much upper body muscle mass in my chest and arms as it is, which really isn't used at all, even with extremely heavy training packs (120-130lbs). If I was to use poles, then oxygen demands would skyrocket beyond my anaerobic threshold and I would hardly move-I have noticed that the demands increase susbtantially when I also carry a 2.2lb dumbell (very little) in each hand as well as close to maximal training loads.
I can certainly understand the use of poles on slippery/unstable terrain, such as sand and snow, where the ground moves substantially under your feet, wasting energy with each step. Here, the addition of two more pressure points will no doubt reduce energy wastage (I hypothesis that as the pressure increases on a ground impact point on eg sand, that energy efficiency decreases exponentially) and therefore be more energy efficient. On hard, normal ground, and particularly trails, I cannot see there being an energy efficiency gain with poles. I can see some possible speed gains, although that is perhaps also debatable-most people probably dont have the VO2 max to fully use their leg muscles to their potential anyway, let alone their upper body aswell-elite runners included.
Note: in South Australia, very few people use poles bushwalking. It nice to take part in a discussion like this :)