I think the reason that people speed climb places like yosemite is that its just plain up more fun that way. Ask Bill Wright or Hans Florine. Florine held the speed record for the Nose with Yuji Hiruyama with just 10 cams and 6 nuts. They finish in 2 hours 48 minutes! (compare that to 45 days? for the original ascent) and is now beaten by the Huber brothers who took 3 minutes off the previous time. Look at Flyin Brian Mcray, featured in the most recent Climbing issue and holder of virtually every El Cap speed record and numerous sub 24 hour first ascents. Many people in the climbing community would say that it is more dangerous, and those climbers also have records like the first 5.14 free solo, but for the average climber routes can be done using systems like simul climbing and short fixing and rapidly take away the boring parts of waiting for belays and so on. As for ultralight, two Pakastani ascents of the Great Trango Azeem ridge with a 30 lb pack, double rack (minus a couple they dropped midway), and the ascent of the Cat's Ear Spire of Karakoram by Eric Decaria and his partner I cant remember with a 12 POUND PACK between them, sleeping in their clothes and a bivy sack. I guess after all the climbing you do in a day, you don't need a puffy sleeping bag to fall asleep it just sort of comes to you alot easier.
Look into Extreme Alpinism, Speed climbing, and psychovertical.com for alot of these ideas, speed climbing can be safe and also very much for the non 5.14 rope guns, it actually works very well in lower difficulties you can climb without great difficulty, just as much as the cutting edge ascents of the past 20 years