Maybe there could be special craft-able items that would be required to carry these items in your inventory in the first place. How do you carry a sword in your backpack without anyone spotting it anyway? This could give the poor man's belt and hunters belt an actual use outside of appearance.
Example A: Germanic iron age warrior

Notice the gear on his belt (including an axe) as well as the scabbard for his sword.
Example B:

Notice the shield across his back while not in use, and his sword in his scabbard again.
It is also worth noting that swords are generally a sidearm but that's a thread for another time.