The initial builder of a palisade, or a person chosen to be the owner, should have about all rights for a piece of wall. My keys have been stolen by 'friends', lost, cornerposts built on the wrong places, as now just one tile to early so its impossible to build a gate there. As owner you should be able to destroy them easily and to get keys of gates if none exist anymore (dropped by someone and decayed).
Sure this needs some thought but like it is now, a random helpful person(/noob) can screw up your whole palisade if you dont look for 1 second.