Removing travelling village idol makes trading 100% more difficult and the same for People joining your village, transporting stuff, travelling in general, its just a complete inconvenience.
Also re-add Travel weariness that way their is a limit to how far you can travel and how much you can do it without drinking wine or sleeping or whatever.