It appears that you can lift object while on horse, but can not drop them, unless you get off the horse.
I do not know whether this is intended, so enjoy the gif.
Relevant giv with demonstration:
Update #1: As I have observed, horses work in a similar manner as boats, only that they also have somewhat of an AI while left alone. Saying that, I propose a suggestion:
Make special sacks that can be attached to tamed (or not) horses which, on right click (when on the horse model) come up with 2 options: take off or access.
The size of the sack may be up to Jorb/Loftar's preferences.
Update #2: This post is suddenly turning into a suggestions update-fest, hence another suggestion:
Rope + Horse + Fence = happy stabled horse