I assume you're going with 3.5, I haven't tried 4 but I haven't heard much about it.
You need one set of dice minimum (one each of d4, d6, d8, d10, d12 and d20), but having more makes things go a lot faster (a set for each player is the best).
You can technically play with just the
SRD, but having the physical books is easier. If you're getting the books, start with Player's Handbook, Dungeon Master's Guide and Monster Manual.
You don't need a board, you'll be drawing your own map layouts on graph paper. Anything will do for figurines, though it helps avoid confusion if you use something resembling what it's supposed to represent. I wouldn't buy the official models, they're far too expensive.
As for setting, I recommend you make your own. Keeping in line with canon in predesigned settings can be cumbersome. You don't have to plan ahead in too much detail, just make a rough sketch of your setting and fill the gaps as you play. If you do want to use a standard high fantasy setting, I guess Forgotten Realms is the default option. I still think it's a bit wasteful to buy setting books, though.