Yes, in heavily used areas, ya gotta pack it out. The climbers' route on Mt. Rainier, for example. Otherwise, when the ranger tells me to pack out all my poop and tp during a 5 or 6 day hike in the Sierra, I solemnly nod my head yes. Then I bury my poop and tp appropriately. I agree that if you bury well, it's not an issue--maybe even beneficial. Besides, bear, deer and HORSES poop freely. No restrictions. (I don't quite understand why horses get a free pass but hikers are required carry out their poop...I digress). Anyway, tp seems very biodegradable; poop is poop. But I have seen tp dug up by bears and scattered near a trail or camp area. So responsible technique is important. In some instances I've burned my tp in a campfire ring. This strikes me as the best thing to do.
But I admit to drawing the line at carrying poop on a multi day hike.