It looks like this is the most divided village ever.
First off - to the second group of people? They can't steal from you. You started building a "village" on their island. There is no island in this game big enough to hold two cities - really none big enough to hold one good sized town i've seen, but definitely nothing big enough to comfortably house two separate governments. You were d-bags for settling on their island. You're not going to get any sympathy. But then from what I've heard from someone who was actually in that village there was a ton of theft going on, some against members of the original village by people who claimed to be in its command structure.
This place is like the hearth equivalent of west virginia. You should rename it deliverance.