Okay, I'm kind of disappointed nobody hasn't pointed this out yet. Ladders. Why ladders? Because it's realistic. It would take 4 boards, 8 blocks, 10 sticks and 10 glue to build a ladder. You put it up against the wall and climb over on the other side of the wall. This would make raiding more realistic and more like what raiding was actually done like back in that time period. You also need two people to carry it, so it needs team work. It has to dry for 20 mins. It does not take damage while you walk with it.
Stupid walls didn't stop raiders a 1000+ years ago, Mongolians scaled the Great Wall of China was ladders and grappling hooks, so I'm pretty sure we can climb over these walls.