This is just a collection of what I've learned from gleanings on the forums + some personal experimenting. I added the info to the wiki as well. No pictures as of yet, may consider adding them at some point. Hopefully this helps, any further tips/information about the system would be appreciated.
Recent data on prospecting distance: Q24 extract prospected a node ~50 diagonal tiles away.
A brief explanation of metal, caves, and depth:
The world consists of at least 6 layers. The first which you'll all have access to is the surface, where all players start. Directly below the surface is the first level of the underground. Caves give you access to this level by creating caverns and tunnels in the ground connected to the surface. However, below that there is at least another 4 levels assuming the underground is built the same as in W4. The level below you can be accessed by building a minehole anywhere on your current level. If you don't have a cave, you can do this on the surface to access lvl 1 of the underground world.
Many new players seem to think you must have a metal node to build a minehole, this is untrue. Mineholes can be built anywhere you can pave a 3x3 square for them. However, if you build a minehole where there is no metal, you will have to snake (a pattern of mining that doesn't require supports) to find metal. This is a horrendously slow process if you don't have high strength and/or good tools. Both of which won't be available to a beginning player.
Metal veins appear randomly in any of the 5 underground layers. They are spawned independently of stone quality and hardness of rock. As such, finding a soft high quality metal node is rare since it takes the overlap of 3 randomly distributed factors. There is no terrain types that are best for prospecting on the surface due to the randomness of how ore is located in the world. The only thing that is guaranteed is that hardness and stone quality go up as you descend in levels. Ore quality is equal to the stone quality of the stone that it is found in. It's also worth noting that metal nodes (at least of tin, copper and iron) tend to be very large. 2000 tiles in total isn't uncommon. Although preferable to build just off to the side of the vein and mine into it, you won't knock out very much of the total vein by building a minehole directly ontop of it if you so wish.
Now that my foreword is done, on to the guide itself:
The prospecting system is at the very least obtuse, and it can quite easily seem broken/bugged to people who are new to prospecting.
When you prospect, you scan an area of ground beneath you for the presence of a metal node. The higher the quality of rustroot extract you use, the more area it will scan. The formula seems to have been changed within recent worlds and is currently unknown, but Q20-30 has found an ore node more than 100 tiles away for me. It is unknown if any skills or attributes are involved in this process.
In preparation for prospecting, you'll want to collect a fairly large amount of rustroot. At least 20 is a good start as it will let you scan around a fair bit with some leftover in case you find a node signal to chase. The quality of extract seems to be dependent on the quality of the root and jar. It is unknown if any skills affect it, and it appears to ignore water quality.
When you scan, if you are standing on an ore vein, it will tell you the ore type that is below you. The next best situation is if there's an ore vein in scanning range, in which case it will give you a dowsing direction to follow. However, when scanning there is a chance that you will fail to detect ore whether or not it is present within scanning range. It seems that once you have failed, a small area (5x5?) around you will no longer give positive readings. If you have gotten on track of a node and fail, move ~5 tiles before scanning again to pick up on the node again. How exactly this failure mechanism works is unknown.
Ender's client is highly suggested for the "show dowsing direction" option that has been added in. This gives a boxed window that will tell you the direction the ore node was in, and unlike the pie slice, it will not fade unless you close it. This allows you a set reference point if you have to leave the area or are trying to get new readings but fail.
When reading Enders dowsing direction information, make sure to note this game is isometric. The center top of your screen is North West, and the top right corner is actually true North. This is annoying because the 0 degree is not based isometrically.
The delta number given doesn't appear to have any bearing on the direction, it may be related to the width of the pie slice. The dial tracks degrees as positive spinning counter clockwise, and negative spinning clockwise. Enders will occasionally give numbers like 203 degrees. If you want the negative number equivalent, simply subtract 360 from the reading. Either way, the pie slice tracking appears to be accurate if you make sure to be paying attention when each prospect attempt finishes.