I've been working on my mansion, and while I agree they should be difficult to build, perhaps the costs could be adjusted. currently, the main difficulty with making a mansion is not the stone, but the thatched roof. This seems odd. Perhaps it should require something like 100 or so straw, and 1000 or so stone, or perhaps it should take a quantity of "any metal ore".
Alternatively, it should require a very expensive "architecture" skill. This would mean that someone would likely have to find someone to place the construction site, making an interesting challenge instead of just another grindy bit.
Just some thoughts. I've just about got there with this wheat thing. I must say, the most fun part about building it was trading for the metal.