I think that clearly some judgment is needed. Yes, it may be arrogant to go about scattering cairns. But it is also arrogant to fill an entire bowl with them because everyone has their favorite route, or just thought they knew best, or in fact were idiots, or whatever. When I find several cairned routes all leading to the same place, yes, I'll start knocking over the more asinine ones. There are areas crisscrossed with social trails because some idiot cairned them, when everyone should have just stuck to the main trail. And stacks of rocks every 10 feet is simply another manifestation of the preponderance of morons in our society. If you really think that knocking over such excessive cairns is wrong then, heck, let's just pave the damned trails.
I'm not going to go about knocking over obviously important cairns. In particular, if someone built a 6-foor cairn I'd be likely to assume that there was a good reason the spot needed marked. Or at a junction, or trailhead, or climb start. But I'll also fight idiocy by destroying the work of idiots, picking up their garbage, etc. Because excessive cairns are just another form of littering, really- almost the equivalent of scratching messages into trees or rock faces. A lot of people don't do them to mark a route- they do them to leave a mark. And I'm smart enough to take seasonal trail conditions into account.
So, both extreme positions are asinine. Unfortunately, most other people lack judgment as good as mine. ;)