Apparently your travel weariness bar is actually calculated by your agi? By this I mean that the bar on the actual debuff picture is relative to your agi rather than being a static amount. This means that if you raise your agi a significant amount and then quick-travel, your weariness will actually go down from the travel rather than up (again, the actual bar image)
I suppose the bug is in the timing. It makes sense that your total weariness would be proportionally lower when you raise your agi, but it should probably make the change immediately rather than wait for your next quick-travel.