I planted a few crops right on the southern edge of my timber house. The crops on either side of my door harvested fine, but the one in front of the door won't harvest, at least not by using the harvest option in the menu. The appearance of the crops is also strange, they appear on the half-paved tile next to the farmland but not on the farmland itself.
since you're trying with the adventure menu, may I suggest instead you try right clicking the plant and selecting harvest. may not work but its atleast worth a try. also doors have a hit box in front of houses, its small but can get in the way at times...so click around.