I agree that fast travel should be heavily limited.
In my opinion let's keep it simple.
- Remove teleport ability to thingwalls entirely. (teleport from thingwall to charter only)
- Make Charter a Realm structure. (only realm members with proper permissions can build one)
- Allow teleportation within same realm only.
Hermists could still easily visit local markets within given realm.
PvP players would have their fun fights when challenging thingwalls, to expand possible teleportation range.
There would actually be a point in expanding realms.