Having been a long time mountain biker, and continuing to this day albeit with a bit of a lapse in my mountain biking activity due to, for a time, living in utterly flat SE Texas, I can say I've seen both sides. It's quite interesting really, to consider now that I never thought I would be much of a hiker in the past but there's actually quite a lot to be said for it obviously. Anyhow, here's some thoughts in random order...
Mountain biking has long since been vilified on account of erosion but it really has a lot to do with a number of factors people often don't talk about, or even really know about so they're all just lumped together. For instance, singletrack ridden in much of the NE US is through dense forest land and many of those trails aren't suitable for human bipedal travel (they're too tight). I've watched trails over decades have zero maintenance with very very little noticeable erosion. On the other end of the spectrum, downhill mountain bikes whose use is limited to a very finite environment (a sizable hill/mountain, with road access to the top) can do a fair amount of damage on switchbacks as people slide around corners at a much greater speed with much more momentum.
However, eroded trails are also no fun to ride and downhillers are extremely likely (more so than your casual XC rider because they're they're the most vilified) to participate in controlled trail building, trail armoring (with rocks), and water-bar creation. And here's the thing, at the end of the day, a single dirt bike, or quad, going up or down any outdoor trail will instantly with one twist of the throttle do more damage than a season of heavy-use by mountain bikers. Moreover, often in the mountainous areas that draw the most mountain bikers, logging and uncontrolled residential development quickly destroys swaths of wild spaces. Strong lobbies back those types of activities however, and mountain biking is an easy scapegoat, especially with outdoor types being pitting against one another while corporations and land developers laugh while we're all in-fighting. Let's not even mention jeep trails and how doubletrack through sensitive national parks blows my mind while I can't take my dog for fear he might take a dump next to a cactus, fern, or kodiak bear that's never experienced dog poo. *rolleyes*
As far as hikers on dedicated trails, sadly I'm rather biased. If you're a hiker and you find a new trail cut by some mountain bikers with jumps all over it, have some common sense and at least avoid sauntering up the ridable line. The good news is that mountain bikers don't go more than several miles into the backcountry, prefer loops, and if you're really out in the middle of nowhere you're not going to find riders, much less jumps and things of that nature (tools are too heavy to carry!). All this said, apparently those wily canadians will lug chainsaws into the backcountry, so British Columbia is an exception to some of these insights....which is why I'm dying to ride British Columbia. But I digress.
Lastly, it's a pretty rare mountain biker that will be rude to a hiker, and a dumb one who will spook a horse. I've known and associated with probably over a hundred riders over the last two decades as a mountain biker and never heard one story of any encounters (from either side of the equation). Near-crashes happen but certainly an adult rider knows we're supposed to yield to everyone. Naturally, YMMV.