I'd love if fast travel got removed, as far as I'm concerned all forms of it.
Pros to removing it heavily outweigh Cons in my opinion.
If you want to keep Charter Stones, make them only accessible from neighbouring thingwalls. And make them require upkeep, 1 rock crystal every week or so to keep them running, so there wouldn't be proxy villages.