The thing here is that it makes sense for armor to be knocked off during a fight with weapons. It makes no sense for a shirt to randomly go flying during a fist fight.