Doubtless dev's have already thought of this, but here's some specific mechanics ideas.
All items and structures have hit points
All items and structures decay over time, items in containers decay slower, containers in buildings decay slower, a stone well for example should easily be able to last 100 in game years without maintenance, a log cabin will fall apart after maybe 20 in game years without maintenance.
Repairing some structures is possible but consumes resources and possibly tools.
Sledge hammers can be used to cause hit point damage to built objects and eventually destroy the object, creating rubble (stone, boards, bits of its creation, general junk). Ideally this should take a long time, possibly multiple days of a single player working constantly to knock down a wooden cabin.
If there's items inside a wooden house when it's destroyed, apply some sort of damage do them and leave what's left.
This works best if houses stay the same size inside as they are outside.