I've gotten most of my basic tools to start off with at Radio Shack, including a soldering iron, desoldering pump
I started out with Shack equipment too, and I've grown to hate their Shack branded irons, and their desoldering pump thing killed several PCBs before I got better equipment. They did sell Weller stuff for a while though, which is good.
I'd agree to start small with most of the tools, but if you're going to do a lot of soldering go ahead and splurge for a good iron. You would not believe how much of a difference it makes. The Hakko FX888 or whatever can usually be had for about a hundo at Amazon. Also those wire tip cleaners are vital to keeping soldering iron tips lasting a long time, some iron holders have them built into the base but they aren't expensive on their own. Much better than using a wet sponge in most cases. Also for the solder, do not get the lead free crap.
Also, get a decent screwdriver set. This is often overlooked by the beginner, but is crucial to keeping your sanity. I normally keep around an inexpensive Black and Decker cordless screwdiver around along with a Walmart (lol) screwdriver set that includes bits that work with the cordless and a precision screwdriver set. This is another thing that I usually buy cheap but not too cheap, as I'm just going to lose half the bits sooner or later.
A few other tools that are useful depending on what you're doing:
Stepper drill bit set, great for putting clean holes in plastic. I use the cheap one you can find at Amazon/Harbor Freight. Works fine for occasional use.
Craftsman Screw Extractor Set- kinda expensive at Sears ($30) but it's been a life saver a couple of times when a tiny screw gets stripped.
Metal File Set - If you're doing any case work you need this for a clean finish. You can start off with one of the cheap kits from Hobby Lobby, and graduate to one of the more expensive sets if you end up doing a lot of work.
Ash tray - I'm serious. It doesn't have to be an ash tray, but keep something around to throw screws into, or they will roll off the desk and into another dimension, never to be seen again.
Gamebits - If you're going to open up old Nintendo systems you're gonna need this. They're only a few bucks, don't cheap out and try the Bic pen thing, it sucks. It can also be helpful to get one of the T-Handle drivers to go with it, some of those screws are stubborn and a regular screwdriver could strip it.