Ah, yes, ultralight backpacking on the cheap. [Homer Simpson voice] Yummmmmm!
Know what you need, then you can pounce on the deal.
Some things are easy. Recycled water bottles and Aquamira or MicroPur for purification. Stanco grease pots for cooking, alcohol or Esbit stoves. Simple little first aid kits, A Swiss Army knife or just a paring knife from the kitchen drawer, a whistle, an LED flashlight. Buy a decent compass, but you don't need to spend $100-- you can get a baseplate rig for $25 or so. Like others said, foam pads are cheap and light and easy to find.
Shelters. Tarps are it. A simple flat tarp is light and inexpensive. It is a good starter sewing project, along with a bivy-- lets call it a sleeping bag cover.
UL packs are actually cheap compared to many of the heavier ones. You'll find production packs like GoLite on eBay and the discount web stores.
Like others said, the shoes gotta fit. But if you are going UL, that means trail runners and they aren't hard to find and there are lots of sales.
Rain gear... ponchos and DriDucks, wait for sales on the good stuff. I don't think expensive rain pants are worth it and you'll tear them up. Red Ledge and Sierra Designs will do the trick with a poncho.
Wicking polyester tees are easy enough to find at discounters and thrift stores. Fleece is cheap and abundant. Zip off pants are on sale all the time and they are an easy find in thrift stores. Running shorts and nylon running pants are easy to find cheap and make very good hiking gear. Socks are pretty cut and dried, but not easy to find good ones cheap. You can use a simple athletic windbreaker until you get The Deal on an UL windshirt.
I've said it before, that UL insulation is the most expensive part of the game. Note how we all went wild when Eddie Bauer has a clearance on their down gear. Insulation is one of the major items to pounce on when the deal comes up.
Good sleeping bags *are* spendy. You'll find a few here on the forums and there will be sales. It depends on the temperature range you need and the climate. You can usually find decent synthetic bags for less than down. Making your own quilt can save you a ton.
Watch the swap forum here and on White Blaze, eBay and GearTrade.com, Campmor, Sierra Trading Post, 6pm.com, and most of the major outdoor web stores have regular sales. The REI outlet can have a zinger now and then, and the REI gear garage in some of their stores can have great deals. If you don't see what you need on the Swap forum, ask-- you never know what people are ready to part with. I am fortunate to live in an area with lots of outdoor activities and the yard sales and thrift stores can be a treasure trove. There are stores that have used gear and take used stuff in trade. Needless to say, I enjoy the hunt :)