1: Require one unit of soil per pot instead of four.
2: Let trees survive on the bench for at least 26 hours. So people who only play for an hour a day have a chance.
Better yet, how about you get rid of the whole "tree pot" thing. Let us put tree seeds/fruits/etc right on a table or in a drying rack and plant the "sprouted" seed right away. Maybe add some chance of failure somewhere but that's it.
What's the worst case scenario for making it easy to plant trees? griefers running around treeing everybody?
Though if such an extreme change was made, I would be fine with trees actually taking longer to grow; as it is, some trees seem to take less time to grow than some crops.