SHAPE: An Unified Approach to Evaluate the Contribution and Cooperation of Individual Modalities

Pengbo Hu, Xingyu Li, Yi ZhouIJCAI 2022 · DOI

TL;DR

This paper introduces SHAPE scores, a novel framework for evaluating how well multi-modal deep learning models utilize different data sources. By leveraging game theory concepts, SHAPE scores quantify both the individual contribution of each modality and the degree of cooperation between them, offering a more nuanced understanding than simple accuracy metrics. This allows researchers to better diagnose model behavior and design more effective multi-modal fusion strategies.

What Problem Does This Paper Address?

Deep learning models are increasingly tasked with processing information from multiple sources, such as text, images, and audio, to tackle complex real-world problems like visual question answering. While researchers have developed sophisticated methods for fusing this multi-modal data, a significant gap exists in understanding how these models actually use the information from each modality. Current evaluation often relies solely on prediction accuracy, which can be misleading. For instance, a model might achieve high accuracy by relying on a single dominant modality while ignoring the potential benefits of others, or it might fail to effectively leverage complementary information. This lack of insight into modality contribution and cooperation hinders the development of truly effective multi-modal systems.

Methods

The core contribution of this paper is the introduction of SHAPE (SHapley value-based PErceptual) scores. These scores are derived from the principles of Shapley values, a concept from cooperative game theory that provides a fair way to distribute payouts among players based on their contributions to a coalition.

In this context:

  • Modalities are treated as "players."
  • Model accuracy on a task is the "utility function" or "payout."

The authors define two types of SHAPE scores:

  1. Marginal Contribution Score ($S_M$): This score quantifies how much a specific modality contributes to the model's performance when added to a coalition of other modalities. It's calculated by averaging the marginal gain of a modality across all possible combinations of other modalities.
  2. Cross-modal Cooperation Score ($C_A$): This score measures the synergy between a set of modalities. It's calculated by comparing the combined contribution of a set of modalities to the sum of their individual marginal contributions. A high cooperation score indicates that the modalities are more than the sum of their parts when used together.

The paper also proposes fine-grained perceptual scores as an extension to existing perceptual scores. These scores aim to disentangle in-class and out-class correlations between modalities, providing a more detailed view of their relationships within the dataset.

Datasets

The authors evaluate their proposed SHAPE scores on three publicly available datasets:

  • B-T4SA: A dataset from Twitter containing text and associated images, used for sentiment analysis. This dataset represents a scenario where modalities are complementary but weakly correlated.
  • CMU-MOSEI: A large dataset for sentence-level sentiment analysis and emotion recognition, consisting of audio, video, and text. For their experiments, they used audio (A) and text (T) modalities, which are complementary and highly correlated.
  • SNLI: A dataset for natural language inference, comprising premise and hypothesis text pairs. This task is presented as one where both modalities are indispensable.

Key Insights and Ideas

  • Accuracy is not enough: The paper strongly argues that prediction accuracy alone is insufficient for evaluating multi-modal models. SHAPE scores reveal that models can achieve high accuracy by exploiting dominant modalities or dataset biases, rather than truly integrating information.
  • Cooperation is often lacking: Even when modalities are complementary, models often fail to exploit cross-modal cooperation, instead relying on a single dominant modality. This is particularly evident in datasets like B-T4SA and CMU-MOSEI.
  • Task and data dependency: The way a model utilizes multi-modal information is heavily influenced by the intrinsic correlation of modalities in the dataset, the nature of the task (e.g., whether modalities are indispensable), and the fusion strategy employed.
  • Fusion strategy matters: The paper suggests that fusion strategies that encourage interaction at earlier stages (like Dense Fusion) are more likely to foster cross-modal cooperation, especially when modalities are indispensable for the task (as seen in SNLI). Late fusion, where information is combined only at the end, often leads to low cooperation scores.
  • Shapley values provide theoretical guarantees: The use of Shapley values offers a theoretically sound foundation for quantifying contributions and cooperation, making the SHAPE scores robust and interpretable.

Key Results

The experimental results across the three datasets highlight several key findings:

  • B-T4SA: For sentiment analysis on B-T4SA, the text modality dominates, with visual modality contributing little. SHAPE scores confirm this, showing near-zero contribution for vision ($S_V$) and low cooperation ($C_{VT}$).
  • CMU-MOSEI: Despite audio and text being complementary and highly correlated, SHAPE scores indicate that the audio modality contributes little to performance, and cooperation is minimal. This suggests models might be exploiting dataset biases or shortcuts.
  • SNLI: In the natural language inference task, where both premise and hypothesis text are crucial, SHAPE scores show comparable contributions from both modalities ($S_{Th}$ and $S_{Tp}$), with hypothesis text being slightly more important. Crucially, cooperation scores ($C_{TpTh}$) are high, indicating that models actively leverage the interaction between the two text inputs. Late fusion models, however, showed significantly lower cooperation.
  • Comparison with Perceptual Scores: The paper demonstrates that SHAPE scores are highly correlated with existing perceptual scores, but SHAPE offers superior interpretability by explicitly separating marginal contribution from cooperation, especially in scenarios with more than two modalities.

Limitations and Future Work

The authors acknowledge that the choice of baseline values for modalities is a critical aspect that deserves more consideration. They suggest that in the future, they will continue to refine the SHAPE scores and systematically evaluate state-of-the-art multi-modal models. They also note that while their SHAPE scores are defined using accuracy as the utility function, they can be readily adapted to other metrics like F1 score.

Summary updated

BibTeX
@inproceedings{hu2022shape,
  author = {Pengbo Hu and Xingyu Li and Yi Zhou},
  title = {{SHAPE: An Unified Approach to Evaluate the Contribution and Cooperation of Individual Modalities}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2022},
  doi = {10.24963/ijcai.2022/425},
}
Figure