I experienced this too building walls (palisade and brick). but other things are fine. And it doesn't seem to effect everyone either.
The ones who see things differently.
You can praise them, disagree with them, quote them, disbelieve them, glorify or vilify them. About the only thing you can't do is ignore them. And while some may see them as the crazy ones, we see genius.