if you don't buy swimming, you're restricted to the North East section of the world. There should probably be a toggle for swimming, in case of accidental clicks and such, I went into deep water trying to gather clay also, but thats just because my mouse has random spaz movements across the monitor.
I'd think linking swimming to your constitution, I mean where you can't even buy the skill until you have the stats to cross a small river, might be a nice idea, but then there are probably better solutions.