I could (and have) done this with garage sale and thrift store gear. As some have noted here, that is a process of opportunity and location, and I spend a fair amount of time on it-- it is a hobby, no doubt about it. Thrift store and garage sale shopping success is directly proportional to time spent.
Next order of bargains: sporting good store sales. These are good for small LED lights, knives and multi-tools, shoes, and sports-oriented clothing like socks, shorts, wicking fabric items like briefs and tees, sunglasses. These stores are very good for running-based gear that transfers so readily to UL hiking. This is where I would go for a cheap sleeping bag and a blue foam pad. Big 5 and GI Joes are examples in my area (Seattle).
Clothing discounters like Burlington Coat Factory, Ross, Marshalls, and TJ Max can offer a lot of wicking sports garments, shoes, socks, briefs, hats, gloves, parkas, rain gear, wind gear and sunglasses. I often see Columbia and simlar brands.
The big box stores like Walmart and Target have a lot of clothing and gadgets. Target has seasonal sales on Gerber items. The Walmart grease pot is well known. I don't have a Walmart close by, so I shop there infrequently, but I did note that they have water bladders cheap.
If you live close to an REI store, the garage sale area is a good place to check. This is where they sell returns and many are "just because" returns. I got a pair of Vasque Velocity shoes for $20 this way.
On the digital side:
Ebay is another area of opportunity and time spent, but the deals are there the range of items is enormous.
Craig's List is the garage sale of the Internet.
There are outlet offerings from most of the major on-line gear retailers: REI, Altrec, Campmor, Backcountry, EMS, Dicks, LL Bean, and others have clearance or sale pages.
On-line tips and caveats: don't be impulsive, know your gear, watch out for knock-offs (North Face is really picked on for this) and make comparisons. I have found a few items on Ebay that were new products to me and a quick search found them to be readily available and for the same or less cost. Shipping costs often negate the bargain, although it can be offset be the convience and driving is getting expensive. I was going to hit the "buy" button on a Maglie LED conversion kit and stopped to research and found that my local REI offered it at the same price, and I didn't have to pay shipping, which came to about 40% of the cost of the actual product. I could afford to wait!
There is a lot of concentration on the cost of the "big three," but the little items add up too. Things like bug juice, sunscreen, Aqua Mira, sunglasses, lighting, compass, whistle, pack towel, knife/multi-tool, first aid, and hygine items can really add up. Clothing is a major cost, and those "intermediate" items like trekking poles can put a strain on the wallet too.
Some of my favorite buys for new gear:
Lafuma sleeping bag: 800 Extreme, GI Joes, $50.
Micro LED lights: Ebay, Campmor, and Big 5, $4-$5 each.
Open Country 1.1 quart aluminum boiler: Campmor, $9.95.
Granite Gear Wisp pack: Ebay, new, $50.
Black Diamond Approach trekking poles: Ebay, new, $30
As far as used stuff, here are some real prices I have paid:
Mountain Hardwear Pack Pants, $12
Jansport external frame pack, $10
Mountain Hardwear polyfill bag (~30F), $10
REI Polayrgaurd HV bag (~20f), $10
Vasque Velocity shoes, $20
Nike Air Wallowa boots, $24
Marmot polyfill sweater $10
REI, Patagonia, Ex-Officio shorts: $2-$4 each
North Face convertible pants, $20
Patagonia, Nike, Under Armour, Hind wicking tops, $3-$4 each
Marmot DriClime windshirt, $12
Nalgene bottles, $2 each
Bolle, Suncloud, Maui Jim, Ray Ban sunglasses, $2 pair
Nike and New Balance wind/running pants $5 pair
Marmot and REI 1/2 zip base layer tops, $5 each
Marmot fleece sweater $10
GoLite Hut1 tarp, $70
Olympus Stylus camera, $3
RidgeRest pads $2-$3 each
REI rucksack (~15OOci) $3
Mittens with fleece liners $2
Fleece beanie $2
Yesterday I bought a pair of Patagonia "Baggy" shorts for $4, a pair of RedLedge rain pants for $2.50, a pair of Speedo Surf Walker water shoes for $1.99 and a Nike runner's wind vest for $2.
The hunt is fun and can be profitable. I've run into all kinds of things I wouldn't have tried if I hadn't come across it at a yard sale or thrift store.