Personally, I find it a little silly that a person can travel just as fast across flat land as they can across a 500 dirt difference tile.
Personally, when traveling across tiles with a certain threshold (Difference between two corners being larger than say, 25-50 dirts) their speed should be capped at something slower, like a maximum of 3 speed.
At maximum, this should slow you to the same crawl speed that a person has when they carry a boat with too many items in it.
And of course, when too much of a dirt difference is present, a cliff is made (As already happens) and people must simply jump.