Interpretable Tensor Fusion

Saurabh Varshneya, Antoine Ledent, Philipp Liznerski, Andriy Balinskyy, Purvanshi Mehta, Waleed Mustafa, Marius Kloft

Abstract

Conventional machine learning methods are predominantly designed to predict outcomes based on a single data type. However, practical applications may encompass data of diverse types, such as text, images, and audio. We introduce interpretable tensor fusion (InTense), a multimodal learning method for training neural networks to simultaneously learn multimodal data representations and their interpretable fusion. InTense can separately capture both linear combinations and multiplicative interactions of diverse data types, thereby disentangling higher-order interactions from the individual effects of each modality. InTense provides interpretability out of the box by assigning relevance scores to modalities and their associations. The approach is theoretically grounded and yields meaningful relevance scores on multiple synthetic and real-world datasets. Experiments on six real-world datasets show that InTense outperforms existing state-of-the-art multimodal interpretable approaches in terms of accuracy and interpretability.

The Challenge of Multimodal Fusion

In multimodal machine learning, the core challenge lies in fusion: effectively combining disparate data types like video, audio, and text into a single predictive model. Most existing models rely on "additive fusion," which simply sums the features of each modality. While easy to interpret, this approach fails to capture complex relationships. For instance, detecting sarcasm often requires identifying the interaction between positive text ("Oh my god! You almost gave me a heart attack") and a specific audio tone or visual smirk.

Conceptual overview of InTense capturing linear and non-linear interactions for sarcasm detection.

Conversely, "Tensor Fusion" models can capture these complex interactions but typically operate as "black boxes." They also suffer from higher-order interaction bias, where models incorrectly attribute importance to complex interactions simply because those mathematical structures have more capacity to fit the data, even when a simpler explanation exists.

Introducing InTense

To address these issues, we developed InTense (Interpretable Tensor Fusion). Built on a generalization of Multiple Kernel Learning (MKL) called Multiple Neural Learning (MNL), InTense offers several key innovations:

  • Joint Learning: It simultaneously optimizes neural representations for each modality and the fusion mechanism itself.
  • Inherent Interpretability: Relevance scores for each modality and their interactions are extracted directly from the fusion layer's weights, providing "out-of-the-box" transparency.
  • Iterative Batch Normalization (IterBN): This technique prevents the model from "cheating" by hiding simple information within complex tensor products. IterBN ensures that the contribution of any subset of modalities is, on average, zero, forcing the model to disentangle unique modality effects from synergistic interactions.

Empirical Results and Insights

InTense was evaluated across a variety of benchmarks, including synthetic datasets (SYNTHGENE) and real-world tasks involving sentiment (CMU-MOSI/MOSEI), humor (UR-FUNNY), and sarcasm (MUStARD).

The "Sarcasm Exception"

A key finding was the "Sarcasm Exception." While text is the dominant modality in most sentiment analysis tasks, InTense revealed that for sarcasm detection, the interaction between modalities is significantly more important than any single modality alone.

Relevance scores across four real-world datasets, highlighting the importance of text (t) in sentiment and interactions (vat) in sarcasm.

Performance and Reliability

InTense outperformed existing interpretable baselines like MRO and MultiRoute. On the MUStARD sarcasm dataset, it achieved 69.6% accuracy, compared to 65.9% for its closest interpretable rival. Furthermore, it performed within 2% of non-interpretable state-of-the-art models like the Multimodal Transformer, proving that transparency does not require significant performance sacrifices. In synthetic tests, InTense was uniquely successful in ignoring "distractor" modalities while accurately identifying non-linear interactions.

Conclusion and Future Directions

InTense provides a theoretically grounded framework for interpretable multimodal learning. By fighting interaction bias and providing reliable relevance scores, it opens the door for safer deployment in critical domains like medical diagnosis or financial assessment. Future work will explore scaling the computational efficiency of tensor products and extending this interpretable fusion to generative tasks.

Summary updated

Abstract

Conventional machine learning methods are predominantly designed to predict outcomes based on a single data type. However, practical applications may encompass data of diverse types, such as text, images, and audio. We introduce interpretable tensor fusion (InTense), a multimodal learning method for training neural networks to simultaneously learn multimodal data representations and their interpretable fusion. InTense can separately capture both linear combinations and multiplicative interactions of diverse data types, thereby disentangling higher-order interactions from the individual effects of each modality. InTense provides interpretability out of the box by assigning relevance scores to modalities and their associations. The approach is theoretically grounded and yields meaningful relevance scores on multiple synthetic and real-world datasets. Experiments on six real-world datasets show that InTense outperforms existing state-of-the-art multimodal interpretable approaches in terms of accuracy and interpretability.
BibTeX
@inproceedings{varshneya2024interpretable,
  author = {Saurabh Varshneya and Antoine Ledent and Philipp Liznerski and Andriy Balinskyy and Purvanshi Mehta and Waleed Mustafa and Marius Kloft},
  title = {{Interpretable Tensor Fusion}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2024},
  url = {https://openreview.net/forum?id=sf8LBVUb64},
}
Figure