There is indeed a trick, and it's called damage soak. The log cabin can (according to the wiki) take 20 damage in one hit and do not lose HP. You could be bonking at it for three days straight and it wouldn't have any effect on the house. That means to actually damage the house you'd have to do at least 21 damage, which means you'd have to have more than 400 strength if you don't use a battering ram.
I'd suggest building a
battering ram and then use it to tear down the house. A battering ram adds 20 damage, which means it would negate the damage soak of the house. Now equip yourself with a stone axe or pickaxe and both use the battering ram at the same time. Shouldn't take too long to bring the house down that way.
The damage you deal while destroying is the square root of your strength with an additional bonus for wielding certain tools, so 16str = 4 damage, 36 str = 6 damage, 144str = 12 damage etc A ram adds a static 20 damage and allows up to 4 people to combine their damage.
The Stone Axe adds an additional 1 damage[citation needed] and the Pick Axe adds an additional 2 damage.