In the world of distributed mobile sensing—think smart homes, autonomous vehicles, or wearable health monitors—we rely on a cocktail of data: video, audio, heart rate, and LiDAR. Multimodal Foundation Models (MFMs) are the current gold standard for making sense of this data because they generalize incredibly well to new tasks. However, they have a "waiting room" problem. Because different sensors produce different data volumes and face varying network conditions, streams arrive at the edge server asynchronously. Standard models usually wait for every single modality to arrive before starting, leading to massive latency spikes.
A new paper, AdaModalX, proposes a way to stop the waiting game by treating multimodal inference as a progressive process rather than a synchronization barrier.
TL;DR
AdaModalX is a progressive inference framework that decouples model execution from modality synchronization. By using lightweight on-device probes and edge-side "contribution assessors," it triggers inference as soon as the available data is sufficient, achieving up to a 64.75% reduction in latency and a 26.97% boost in accuracy over traditional methods.
What Problem Does This Paper Address?
The authors tackle modality asynchrony. In a real-world distributed system, an audio clip might arrive at the server in milliseconds, while a high-resolution video frame is still chugging through a congested Wi-Fi link.
Existing systems face a lose-lose trade-off:
- Wait for everything: This ensures accuracy but kills real-time responsiveness (high latency).
- Run on what's there: This is fast but often leads to poor accuracy because the model is missing critical context (the "incomplete input" problem).
AdaModalX argues that we shouldn't care about completeness; we should care about utility. If the audio stream alone is enough to identify a "glass breaking" event with high confidence, why wait for the video?
Methods
AdaModalX introduces a two-stage "Progressive Data Orchestration" pipeline:
-
Importance-Aware Data Uploading (Device-Side): Instead of blindly offloading all data, each mobile device runs a lightweight "probe" distilled from the large edge-side MFM. This probe uses a joint confidence rule (looking at both the top prediction score and the margin between the first and second best guesses). If the device is certain, it exits locally. If not, it uploads the data.
-
On-Demand MFM Activation (Edge-Side): The edge server doesn't just sit idle. It uses a lightweight contribution assessor to evaluate the "utility" of the data that has arrived.
-
Feature Imputation: If the arrived data isn't quite enough, AdaModalX doesn't wait for the rest of the raw data. Instead, it uses a cross-modal context token to "impute" (estimate) the missing features from the partially received delayed inputs.
- Selective Activation: The full MFM is only triggered once the combined utility of arrived and imputed features hits a specific threshold.
Datasets
The researchers evaluated AdaModalX across four diverse tasks using two MFM architectures (ImageBind and Qwen3.5-2B):
- Event Detection ($T_1$): Using the AVE dataset (audio-visual event localization).
- Driver Activity Recognition ($T_2$): Using the Drive&Act dataset (multi-view, multi-modal driver behavior).
- Human Activity Recognition ($T_3$): Using the Kinetics benchmark (large-scale video-audio).
- Human Sentiment Analysis ($T_4$): Using CMU-MOSI (aligned text, audio, and visual modalities).
Key Insights and Ideas
- Utility > Completeness: The core "aha moment" is that modalities contribute unequally. For many samples, a subset of data provides 99% of the necessary information. AdaModalX formalizes how to measure that "incremental utility."
- Feature-Level over Raw-Level: Previous attempts at handling missing data tried to synthesize raw pixels or audio (which is computationally heavy and error-prone). AdaModalX performs imputation in the shared semantic embedding space, which is much more robust to noise.
- Distillation for Confidence: The device-side probes aren't just smaller versions of the MFM; they are specifically trained via distillation to mimic the MFM’s decision boundaries, ensuring that "early exits" on the device are reliable.
Key Results
The experiments show that AdaModalX effectively pushes the Pareto frontier of the accuracy-latency trade-off:
- Latency Reduction: Achieved up to 64.75% lower end-to-end latency compared to synchronous baselines.
- Accuracy Improvement: Improved accuracy by up to 26.97% compared to methods that simply run on incomplete data without imputation.
- Bandwidth Robustness: In scenarios with fluctuating network speeds (2Mbps to 50Mbps), AdaModalX maintained high accuracy while other methods saw performance collapses or massive delays.
- Efficiency: The edge-side orchestration modules (assessor and imputer) add negligible overhead—only about 1.38% of the total MFM latency.
Limitations and Future Work
While impressive, the authors note a few areas for growth:
- Large-Scale Deployment: The current evaluation primarily uses emulated asynchrony under controlled network traces. Testing in massive, "wild" live deployments is the next step.
- Architecture Diversity: The framework was tested on two major MFMs, but the authors aim to extend it to a wider variety of foundation model architectures and even more heterogeneous sensor modalities (like radar or specialized medical sensors).
- Dynamic Retraining: Currently, the contribution assessor is trained offline. Making this assessor adapt online to shifting data distributions would make the system even more resilient.
Paper Citation: Xiangrui Xu, Sicong Liu, Zhiwen Yu, Zimu Zhou, and Bin Guo. "AdaModalX: Synchronization-Free MFM Inference for Distributed Multimodal Mobile Sensing via Progressive Data Orchestration." arXiv:2611.20464v1 [cs.LG], 2026.
Summary updated