Multimodal machine learning has traditionally been a "Big Three" game, dominated by language, vision, and audio. But the real world is far messier, filled with proprioception sensors on robots, tabular medical records, and high-frequency time-series data. Scaling models to handle this "high-modality" reality usually leads to a parameter explosion—until now.
TL;DR
HighMMT is a new framework that scales multimodal representation learning to 10 diverse modalities by quantifying "heterogeneity"—the degree to which different data types encode unique information. By using information-theoretic metrics to decide when to share or separate model parameters, HighMMT achieves state-of-the-art performance across 15 tasks while using up to 10x fewer parameters than specialized models.
What Problem Does This Paper Address?
As we move toward general-purpose AI, we need models that can ingest any type of data. However, the current approach is unscalable: adding a new, dedicated encoder for every new sensor or data format is prohibitively expensive in terms of memory and computation.
The core challenge is heterogeneity. How do we know if a model trained on "images" can share its knowledge with a model processing "robot force sensors"? If two modalities are similar, we should share parameters to gain statistical strength; if they are vastly different, sharing might lead to "negative transfer," where learning one task hurts performance on another. The authors seek a principled way to quantify these relationships to build efficient, high-modality transformers.
Methods
The authors introduce a two-step workflow for building the High-Modality Multimodal Transformer (HighMMT):
1. Heterogeneity Quantification
The researchers propose two new metrics based on modality information transfer:
- Modality Heterogeneity: Measures how much information transfers from modality $X_1$ to $X_2$ by comparing a model trained on $X_2$ from scratch vs. one fine-tuned from $X_1$.
- Interaction Heterogeneity: Measures how similarly pairs of modalities (e.g., {Audio, Text} vs. {Video, Text}) interact when fused.
2. The HighMMT Architecture
HighMMT uses a standardized pipeline to handle diverse inputs:
- Serialization: All modalities (images, tables, sets) are converted into a sequence of embeddings.
- Shared Unimodal Encoder: A Perceiver-based architecture serves as a modality-agnostic backbone.
- Crossmodal Transformers: These capture pairwise interactions between modalities.
- Dynamic Parameter Sharing: Instead of full sharing or no sharing, HighMMT uses the heterogeneity metrics to cluster modalities. Similar modalities share the same "parameter group," while outliers get their own.
Datasets
The study utilizes MultiBench, a comprehensive benchmark for multimodal learning. The experiments span:
- 10 Modalities: Text, image, audio, video, sensors, proprioception, speech, time-series, sets, and tables.
- 15 Tasks: Ranging from sentiment analysis (MOSEI) and humor detection (UR-FUNNY) to mortality prediction (MIMIC) and robotic object pose estimation (Push).
- 5 Research Areas: Affective computing, healthcare, robotics, multimedia, and human-computer interaction (HCI).
Key Insights and Ideas
- The Low-Rank Shortcut: One might worry that measuring heterogeneity for every pair of modalities is too expensive ($O(M^2)$). However, the authors discovered that the heterogeneity matrix is low-rank. A rank-3 approximation is enough to capture the relationships, meaning we only need to sample a few pairs to estimate the whole matrix.
- Interactions are Unique: The study found that interaction heterogeneity is generally higher than unimodal heterogeneity. In plain English: while many modalities might look similar to an encoder, the way they combine to solve a task is often highly specific.
- Cross-Modal Generalization: Pre-training on one set of modalities (like vision and audio) actually improves performance when fine-tuning on completely different ones (like tabular medical data). This suggests the model learns a "universal" way to process structured information.
Key Results
HighMMT pushes the Pareto front of the performance-efficiency tradeoff, outperforming over 10,000 possible combinations of task-specific models.
- Efficiency: HighMMT achieves results comparable to or better than state-of-the-art (SOTA) specialized models while using 8x to 10x fewer total parameters.
- Scaling: Unlike many models that plateau, HighMMT’s performance continues to improve as more modalities are added to the multitask training set.
- Low-Resource Boost: The model is particularly effective for "small" datasets. For example, on the UR-FUNNY dataset, multitask pre-training provided a +2.4% accuracy boost compared to training on that modality alone.
Limitations and Future Work
While HighMMT is a major step forward, the authors note a few areas for growth:
- Initial Overhead: Even with low-rank approximations, the initial "probing" phase to calculate heterogeneity requires training several small models.
- Metric Assumptions: The metrics rely on a "relaxed triangle inequality" (if A is like B, and B is like C, then A is somewhat like C). While this held true 96% of the time in their tests, it may not hold for all future modalities.
- Beyond 10 Modalities: While 10 is "high" by today's standards, future robotic systems may have hundreds of specialized sensors, which will further test the limits of this clustering approach.
HighMMT provides a blueprint for a more "universal" multimodal AI—one that doesn't just see and hear, but senses the world through every available digital straw.
Summary updated