High-Modality Multimodal Transformer: Quantifying Modality & Interaction Heterogeneity for High-Modality Representation Learning

Paul Pu Liang, Yiwei Lyu, Xiang Fan, Jeffrey Tsaw, Yudong Liu, Shentong Mo, Dani Yogatama, Louis-Philippe Morency, Ruslan Salakhutdinov

Multimodal machine learning has traditionally been a "Big Three" game, dominated by language, vision, and audio. But the real world is far messier, filled with proprioception sensors on robots, tabular medical records, and high-frequency time-series data. Scaling models to handle this "high-modality" reality usually leads to a parameter explosion—until now.

TL;DR

HighMMT is a new framework that scales multimodal representation learning to 10 diverse modalities by quantifying "heterogeneity"—the degree to which different data types encode unique information. By using information-theoretic metrics to decide when to share or separate model parameters, HighMMT achieves state-of-the-art performance across 15 tasks while using up to 10x fewer parameters than specialized models.

What Problem Does This Paper Address?

As we move toward general-purpose AI, we need models that can ingest any type of data. However, the current approach is unscalable: adding a new, dedicated encoder for every new sensor or data format is prohibitively expensive in terms of memory and computation.

The core challenge is heterogeneity. How do we know if a model trained on "images" can share its knowledge with a model processing "robot force sensors"? If two modalities are similar, we should share parameters to gain statistical strength; if they are vastly different, sharing might lead to "negative transfer," where learning one task hurts performance on another. The authors seek a principled way to quantify these relationships to build efficient, high-modality transformers.

Methods

The authors introduce a two-step workflow for building the High-Modality Multimodal Transformer (HighMMT):

1. Heterogeneity Quantification

The researchers propose two new metrics based on modality information transfer:

  • Modality Heterogeneity: Measures how much information transfers from modality $X_1$ to $X_2$ by comparing a model trained on $X_2$ from scratch vs. one fine-tuned from $X_1$.
  • Interaction Heterogeneity: Measures how similarly pairs of modalities (e.g., {Audio, Text} vs. {Video, Text}) interact when fused.
The HighMMT workflow: estimating heterogeneity, clustering parameters, and training the aware model.

2. The HighMMT Architecture

HighMMT uses a standardized pipeline to handle diverse inputs:

  • Serialization: All modalities (images, tables, sets) are converted into a sequence of embeddings.
  • Shared Unimodal Encoder: A Perceiver-based architecture serves as a modality-agnostic backbone.
  • Crossmodal Transformers: These capture pairwise interactions between modalities.
  • Dynamic Parameter Sharing: Instead of full sharing or no sharing, HighMMT uses the heterogeneity metrics to cluster modalities. Similar modalities share the same "parameter group," while outliers get their own.

Datasets

The study utilizes MultiBench, a comprehensive benchmark for multimodal learning. The experiments span:

  • 10 Modalities: Text, image, audio, video, sensors, proprioception, speech, time-series, sets, and tables.
  • 15 Tasks: Ranging from sentiment analysis (MOSEI) and humor detection (UR-FUNNY) to mortality prediction (MIMIC) and robotic object pose estimation (Push).
  • 5 Research Areas: Affective computing, healthcare, robotics, multimedia, and human-computer interaction (HCI).

Key Insights and Ideas

  • The Low-Rank Shortcut: One might worry that measuring heterogeneity for every pair of modalities is too expensive ($O(M^2)$). However, the authors discovered that the heterogeneity matrix is low-rank. A rank-3 approximation is enough to capture the relationships, meaning we only need to sample a few pairs to estimate the whole matrix.
  • Interactions are Unique: The study found that interaction heterogeneity is generally higher than unimodal heterogeneity. In plain English: while many modalities might look similar to an encoder, the way they combine to solve a task is often highly specific.
  • Cross-Modal Generalization: Pre-training on one set of modalities (like vision and audio) actually improves performance when fine-tuning on completely different ones (like tabular medical data). This suggests the model learns a "universal" way to process structured information.

Key Results

HighMMT pushes the Pareto front of the performance-efficiency tradeoff, outperforming over 10,000 possible combinations of task-specific models.

  • Efficiency: HighMMT achieves results comparable to or better than state-of-the-art (SOTA) specialized models while using 8x to 10x fewer total parameters.
  • Scaling: Unlike many models that plateau, HighMMT’s performance continues to improve as more modalities are added to the multitask training set.
  • Low-Resource Boost: The model is particularly effective for "small" datasets. For example, on the UR-FUNNY dataset, multitask pre-training provided a +2.4% accuracy boost compared to training on that modality alone.
Overall tradeoff showing HighMMT pushing the Pareto front of performance vs. efficiency.

Limitations and Future Work

While HighMMT is a major step forward, the authors note a few areas for growth:

  • Initial Overhead: Even with low-rank approximations, the initial "probing" phase to calculate heterogeneity requires training several small models.
  • Metric Assumptions: The metrics rely on a "relaxed triangle inequality" (if A is like B, and B is like C, then A is somewhat like C). While this held true 96% of the time in their tests, it may not hold for all future modalities.
  • Beyond 10 Modalities: While 10 is "high" by today's standards, future robotic systems may have hundreds of specialized sensors, which will further test the limits of this clustering approach.

HighMMT provides a blueprint for a more "universal" multimodal AI—one that doesn't just see and hear, but senses the world through every available digital straw.

Summary updated

Abstract

Many real-world problems are inherently multimodal, from spoken language, gestures, and paralinguistics humans use to communicate, to force, proprioception, and visual sensors on robots. While there has been an explosion of interest in multimodal learning, these methods are focused on a small set of modalities primarily in language, vision, and audio. In order to accelerate generalization towards diverse and understudied modalities, this paper studies efficient representation learning for high-modality scenarios involving a large set of diverse modalities. Since adding new models for every new modality becomes prohibitively expensive, a critical technical challenge is heterogeneity quantification: how can we measure which modalities encode similar information and interactions in order to permit parameter sharing with previous modalities? This paper proposes two new information theoretic metrics for heterogeneity quantification: (1) modality heterogeneity studies how similar 2 modalities {X1,X2} are by measuring how much information can be transferred from X1 to X2, while (2) interaction heterogeneity studies how similarly pairs of modalities {X1,X2}, {X3,X4} interact by measuring how much information can be transferred from fusing {X1,X2} to {X3,X4}. We show the importance of these 2 proposed metrics as a way to automatically prioritize the fusion of modalities that contain unique information or interactions. The result is a single model, HighMMT, that scales up to 10 modalities (text, image, audio, video, sensors, proprioception, speech, time-series, sets, and tables) and 15 tasks from 5 research areas. Not only does HighMMT outperform prior methods on the tradeoff between performance and efficiency, it also demonstrates a crucial scaling behavior: performance continues to improve with each modality added, and it transfers to entirely new modalities and tasks during fine-tuning.
BibTeX
@article{liang2023high,
  author = {Paul Pu Liang and Yiwei Lyu and Xiang Fan and Jeffrey Tsaw and Yudong Liu and Shentong Mo and Dani Yogatama and Louis-Philippe Morency and Ruslan Salakhutdinov},
  title = {{High-Modality Multimodal Transformer: Quantifying Modality \& Interaction Heterogeneity for High-Modality Representation Learning}},
  journal = {Trans. Mach. Learn. Res.},
  year = {2023},
}
Figure