Shelter: Try something like a Tarptent Contrail if you are in love with the idea of sleeping under a tarp, or you could just "rip the bandaid off all at once" and get yourself a one person tarp, and a bivy sack for a decent price from companies like TiGoat and Oware that have pretty reasonably priced gear.
Sleeping bags: They are very expensive, but Western Mountaineering sleeping bags are some of the lightest and the best sleeping bags on the market. Ebay can help relieve some of the effect on your wallet. Marmot has some pretty good lightweight bags too (Atom, Hydrogen, Helium).
Footwear: Montrail has been a dependable source of trail runners over the years, and recently brands like GoLite and Inov-8 have his the market with shoes as well. Try going to a local running store and getting fitted up with a decent pair of trail runners, you can always go online and find them for much cheaper online.
Packs: GoLite makes very good lightweight backpacks, the Jam would be a very good intro. Also check out smaller brands like ULA.
Misc.: For a stove, try a very small canister stove and a small aluminum/titanium pot. Another very important aspect of having a lighter backpack is to do a good overhaul of your clothing closest. Companies like GoLite, Montbell, and even Marmot have some very good lightweight clothes.
Look at all of the Reader Reviews that this site has to offer to get an idea of the brands and concepts of packing lighter weight gear