I've broken carbon fiber poles and aluminum poles. In all cases, it was because of misuse. I can put all of my weight on GG poles (and often do) and it is no problem. The biggest difference in durability is when you step on it, wedge it, or otherwise put pressure on the middle of the pole. Metal poles can take a lot of force before they bend. Once they bend, they can still be used for a while (long enough to get you through a trip). The same can't be said about carbon fiber. Like I said, though, I've seen plenty of metal poles that are bent (and ultimately broken). That is why I use the tougher, heavier metal poles in the winter (when I'm likely to fall on them). It is also why Black Diamond makes different types of metal poles (some are really tough, some aren't).
I also use the poles to support my tent. On my last trip, I stupidly stepped on my pole and broke it. Fortunately, my tent can be supported with one pole. Even without that, though, I would have been OK. There were plenty of sticks and trees around. I probably could have also splinted my pole, as well. One thing to consider (if you don't want to worry about it) is to simply carry a 2 ounce pole for the tent (a spare pole, if you will). Tarp Tent sells these: (http://tinyurl.com/447hxjw). Your overall weight will be much lower than it is now. As everyone said, you will really notice the difference in the pole weight (unlike back weight). The biggest negative is the price.
As to the original question, swing weight is also a factor, but that gets complicated. Of the poles mentioned, the swing weight is roughly equivalent to the total weight (since all the poles are the same top to bottom).