Villages work should go both ways, me to you and you to me. From the OP we could only see that you build the wall, and people take care for their own things + pay rent. That means you had to do one time investment into building wall (time+materials), while they will need to pay for their life permanently, which isn't that good.
You could for example change your offer to I give you place to stay and in return you bring me some of your first curios you can find, then we can work in the town together, you will farm, I will mine, 3rd person will do silk, 4th will do animals etc.