Caves are annoying to navigate manually since if you touch a cave wall, even at an angle, your movement is immediately halted. This makes walking through caves more tedious than it is on the surface. This behavior is different from other walls, such as palisades or fences, where your character will slide along the wall if you bump into it. We should have the same for caves' walls.
Caves are already annoying to navigate in that you might have narrow tunnels that you want to get through. They don't need this extra layer of annoyance.