It's a display bug with the table, it doesn't seem to check the timer that starts when the trees sprout (and kills them when it runs out). Opening the table refreshes the data and it'll notice that the pots have been neglected for to long and they died.
You have to plant them from the pots in time. Same with silkworms, leave them for to long and they'll vanish.