While many objects would simply be able to be destroyed by hand or by weapon, others taking a very very long time.
I believe that stronger structures and objects such as chests should not be destructible by hand.
Introducing, SEIGE MACHINES! Slow moving structures available only to towns people requiring roughly just as many materials as a town totem - Wide and unable to cross clay or swamp - This would make constructing roads or bridges towards enemy fortresses a real tactical goal. And heres the REAL kicker
Seige machines should only be constructible within the confines of the town claim - meaning you'd have to haul this HUGE machine, and keep it guarded in order to infiltrated and destroy enemy areas.
I believe this may make things interesting - No simple griefers could ever achieve such an accomplishment