If houses had keys, required rams (or high strength) to break and had drying times (easily breakable before they dry, so you can't trap someone with them) then walls would be much less of a necessity for 80% of the population.
You could even just remove the concept of houses altogether and instead of building houses you could build house-walls which would be built like palisades and when closed off would become what we know know as houses, therefore allowing for customizable house sizes while also getting rid of pocket worlds.