Modality Complementariness: Towards Understanding Multi-modal Robustness
Lists
Only you see this
Abstract
Along with the success of multi-modal learning, the robustness of multi-modal learning is receiving attention due to real-world safety concerns. Multi-modal models are anticipated to be more robust due to the possible redundancy between modalities. However, some empirical results have offered contradictory conclusions. In this paper, we point out an essential factor that causes this discrepancy: The difference in the amount of modality-wise complementary information. We provide an information-theoretical analysis of how the modality complementariness affects the multi-modal robustness. Based on the analysis, we design a metric for quantifying how complementary the modalities are to others and propose an effective pipeline to calculate our metric. Experiments on carefully-designed synthetic data verify our theory. Further, we apply our metric to real-world multi-modal datasets and reveal their property. To our best knowledge, we are the first to identify modality complementariness as an important factor affecting multi-modal robustness.
BibTeX
@inproceedings{li2023modality,
author = {Siting Li and Chenzhuang Du and Yu Huang and Longbo Huang and Hang Zhao},
title = {{Modality Complementariness: Towards Understanding Multi-modal Robustness}},
booktitle = {International Conference on Learning Representations},
year = {2023},
url = {https://openreview.net/forum?id=gfHLOC35Zh},
}Loading related content...