A couple of issues I see with tiles that would be near impossible to mine or completely unmineable are I foresee the possibility of someone creating a minehole down in the middle of such an area in a way that their surrounding location is comprised of tiles that are close to or totally unmineable thus creating a vault scenario.
Second one would be that if you do create such a build-near limitation this won't matter if you don't actually need any walls (unmineable tiles) and it's also just an annoying implementation.