When items are averaged, they're averaged as integers, so any remainder is dropped, not rounded. This should be changed in the next update (according to post by jorb in the last update). If you put in anything that lowers one of the quality values so it goes down, it drops the quality instead of being averaged out. It quickly becomes a downward spiral.
Until that change occurs, only put fresh fodder into an empty trough. If need be, use buckets to empty the old one.
Also to note, this is the way it has always worked if you wanted to get any real increases to quality. It was easy to empty out troughs, though, by picking one up and right clicking another (assuming that is still a thing, but haven't gotten any livestock yet).