The right-clicking only works for villages that were made inactive by a recent server corruption (As far as I understand it).
I believe you are saying the original village idol has been destroyed completely. In which case I believe the remaining flags will have to be destroyed and there is no way to recover the materials.