Abstract
Conventional machine learning methods are predominantly designed to predict outcomes based on a single data type. However, practical applications may encompass data of diverse types, such as text, images, and audio. We introduce interpretable tensor fusion (InTense), a multimodal learning method for training neural networks to simultaneously learn multimodal data representations and their interpretable fusion. InTense can separately capture both linear combinations and multiplicative interactions of diverse data types, thereby disentangling higher-order interactions from the individual effects of each modality. InTense provides interpretability out of the box by assigning relevance scores to modalities and their associations. The approach is theoretically grounded and yields meaningful relevance scores on multiple synthetic and real-world datasets. Experiments on six real-world datasets show that InTense outperforms existing state-of-the-art multimodal interpretable approaches in terms of accuracy and interpretability.
The Challenge of Multimodal Fusion
In multimodal machine learning, the core challenge lies in fusion: effectively combining disparate data types like video, audio, and text into a single predictive model. Most existing models rely on "additive fusion," which simply sums the features of each modality. While easy to interpret, this approach fails to capture complex relationships. For instance, detecting sarcasm often requires identifying the interaction between positive text ("Oh my god! You almost gave me a heart attack") and a specific audio tone or visual smirk.
Conversely, "Tensor Fusion" models can capture these complex interactions but typically operate as "black boxes." They also suffer from higher-order interaction bias, where models incorrectly attribute importance to complex interactions simply because those mathematical structures have more capacity to fit the data, even when a simpler explanation exists.
Introducing InTense
To address these issues, we developed InTense (Interpretable Tensor Fusion). Built on a generalization of Multiple Kernel Learning (MKL) called Multiple Neural Learning (MNL), InTense offers several key innovations:
- Joint Learning: It simultaneously optimizes neural representations for each modality and the fusion mechanism itself.
- Inherent Interpretability: Relevance scores for each modality and their interactions are extracted directly from the fusion layer's weights, providing "out-of-the-box" transparency.
- Iterative Batch Normalization (IterBN): This technique prevents the model from "cheating" by hiding simple information within complex tensor products. IterBN ensures that the contribution of any subset of modalities is, on average, zero, forcing the model to disentangle unique modality effects from synergistic interactions.
Empirical Results and Insights
InTense was evaluated across a variety of benchmarks, including synthetic datasets (SYNTHGENE) and real-world tasks involving sentiment (CMU-MOSI/MOSEI), humor (UR-FUNNY), and sarcasm (MUStARD).
The "Sarcasm Exception"
A key finding was the "Sarcasm Exception." While text is the dominant modality in most sentiment analysis tasks, InTense revealed that for sarcasm detection, the interaction between modalities is significantly more important than any single modality alone.
Performance and Reliability
InTense outperformed existing interpretable baselines like MRO and MultiRoute. On the MUStARD sarcasm dataset, it achieved 69.6% accuracy, compared to 65.9% for its closest interpretable rival. Furthermore, it performed within 2% of non-interpretable state-of-the-art models like the Multimodal Transformer, proving that transparency does not require significant performance sacrifices. In synthetic tests, InTense was uniquely successful in ignoring "distractor" modalities while accurately identifying non-linear interactions.
Conclusion and Future Directions
InTense provides a theoretically grounded framework for interpretable multimodal learning. By fighting interaction bias and providing reliable relevance scores, it opens the door for safer deployment in critical domains like medical diagnosis or financial assessment. Future work will explore scaling the computational efficiency of tensor products and extending this interpretable fusion to generative tasks.
Summary updated