AdaModalX: Synchronization-Free MFM Inference for Distributed Multimodal Mobile Sensing via Progressive Data Orchestration

Xiangrui Xu, Sicong Liu, Zhiwen Yu, Zimu Zhou, Bin GuoarXiv 2026 · DOI · arXiv

In the world of distributed mobile sensing—think smart homes, autonomous vehicles, or wearable health monitors—we rely on a cocktail of data: video, audio, heart rate, and LiDAR. Multimodal Foundation Models (MFMs) are the current gold standard for making sense of this data because they generalize incredibly well to new tasks. However, they have a "waiting room" problem. Because different sensors produce different data volumes and face varying network conditions, streams arrive at the edge server asynchronously. Standard models usually wait for every single modality to arrive before starting, leading to massive latency spikes.

A new paper, AdaModalX, proposes a way to stop the waiting game by treating multimodal inference as a progressive process rather than a synchronization barrier.

TL;DR

AdaModalX is a progressive inference framework that decouples model execution from modality synchronization. By using lightweight on-device probes and edge-side "contribution assessors," it triggers inference as soon as the available data is sufficient, achieving up to a 64.75% reduction in latency and a 26.97% boost in accuracy over traditional methods.

What Problem Does This Paper Address?

The authors tackle modality asynchrony. In a real-world distributed system, an audio clip might arrive at the server in milliseconds, while a high-resolution video frame is still chugging through a congested Wi-Fi link.

Existing systems face a lose-lose trade-off:

  1. Wait for everything: This ensures accuracy but kills real-time responsiveness (high latency).
  2. Run on what's there: This is fast but often leads to poor accuracy because the model is missing critical context (the "incomplete input" problem).

AdaModalX argues that we shouldn't care about completeness; we should care about utility. If the audio stream alone is enough to identify a "glass breaking" event with high confidence, why wait for the video?

Methods

AdaModalX introduces a two-stage "Progressive Data Orchestration" pipeline:

  1. Importance-Aware Data Uploading (Device-Side): Instead of blindly offloading all data, each mobile device runs a lightweight "probe" distilled from the large edge-side MFM. This probe uses a joint confidence rule (looking at both the top prediction score and the margin between the first and second best guesses). If the device is certain, it exits locally. If not, it uploads the data.

  2. On-Demand MFM Activation (Edge-Side): The edge server doesn't just sit idle. It uses a lightweight contribution assessor to evaluate the "utility" of the data that has arrived.

  3. Feature Imputation: If the arrived data isn't quite enough, AdaModalX doesn't wait for the rest of the raw data. Instead, it uses a cross-modal context token to "impute" (estimate) the missing features from the partially received delayed inputs.

  4. Selective Activation: The full MFM is only triggered once the combined utility of arrived and imputed features hits a specific threshold.

Datasets

The researchers evaluated AdaModalX across four diverse tasks using two MFM architectures (ImageBind and Qwen3.5-2B):

  • Event Detection ($T_1$): Using the AVE dataset (audio-visual event localization).
  • Driver Activity Recognition ($T_2$): Using the Drive&Act dataset (multi-view, multi-modal driver behavior).
  • Human Activity Recognition ($T_3$): Using the Kinetics benchmark (large-scale video-audio).
  • Human Sentiment Analysis ($T_4$): Using CMU-MOSI (aligned text, audio, and visual modalities).

Key Insights and Ideas

  • Utility > Completeness: The core "aha moment" is that modalities contribute unequally. For many samples, a subset of data provides 99% of the necessary information. AdaModalX formalizes how to measure that "incremental utility."
  • Feature-Level over Raw-Level: Previous attempts at handling missing data tried to synthesize raw pixels or audio (which is computationally heavy and error-prone). AdaModalX performs imputation in the shared semantic embedding space, which is much more robust to noise.
  • Distillation for Confidence: The device-side probes aren't just smaller versions of the MFM; they are specifically trained via distillation to mimic the MFM’s decision boundaries, ensuring that "early exits" on the device are reliable.

Key Results

The experiments show that AdaModalX effectively pushes the Pareto frontier of the accuracy-latency trade-off:

  • Latency Reduction: Achieved up to 64.75% lower end-to-end latency compared to synchronous baselines.
  • Accuracy Improvement: Improved accuracy by up to 26.97% compared to methods that simply run on incomplete data without imputation.
  • Bandwidth Robustness: In scenarios with fluctuating network speeds (2Mbps to 50Mbps), AdaModalX maintained high accuracy while other methods saw performance collapses or massive delays.
  • Efficiency: The edge-side orchestration modules (assessor and imputer) add negligible overhead—only about 1.38% of the total MFM latency.

Limitations and Future Work

While impressive, the authors note a few areas for growth:

  • Large-Scale Deployment: The current evaluation primarily uses emulated asynchrony under controlled network traces. Testing in massive, "wild" live deployments is the next step.
  • Architecture Diversity: The framework was tested on two major MFMs, but the authors aim to extend it to a wider variety of foundation model architectures and even more heterogeneous sensor modalities (like radar or specialized medical sensors).
  • Dynamic Retraining: Currently, the contribution assessor is trained offline. Making this assessor adapt online to shifting data distributions would make the system even more resilient.

Paper Citation: Xiangrui Xu, Sicong Liu, Zhiwen Yu, Zimu Zhou, and Bin Guo. "AdaModalX: Synchronization-Free MFM Inference for Distributed Multimodal Mobile Sensing via Progressive Data Orchestration." arXiv:2611.20464v1 [cs.LG], 2026.

Summary updated

Abstract

Multimodal foundation models (MFMs) support generalizable inference in distributed mobile sensing applications by integrating heterogeneous data streams. However, existing MFM inference pipelines typically require complete and synchronized multimodal inputs, whereas mobile sensing streams arrive asynchronously due to network variability and heterogeneous data volumes across modalities. Waiting for all modalities increases end-to-end latency, while inference with incomplete inputs can reduce accuracy. We argue that multimodal inference should depend on the utility of available information rather than modality completeness. Based on this principle, we propose AdaModalX, a progressive multimodal inference framework that decouples inference execution from modality synchronization through runtime utility-aware orchestration. At each device, importance-aware data uploading uses a lightweight probe distilled from the edge-side MFM to allow reliable samples to exit locally while uncertain samples continue uploading. At the edge, on-demand MFM activation constructs candidates from fully received samples and, when needed, imputed features derived from partially received delayed inputs. A lightweight contribution assessor built on frozen MFM encoders estimates candidate utility and activates the MFM once the candidate meets its corresponding utility threshold. Experiments on four mobile sensing applications show that AdaModalX improves the accuracy-latency trade-off, achieving up to 26.97% accuracy improvement and up to 64.75% end-to-end latency reduction.
BibTeX
@misc{xu2026adamodalx,
  author = {Xiangrui Xu and Sicong Liu and Zhiwen Yu and Zimu Zhou and Bin Guo},
  title = {{AdaModalX: Synchronization-Free MFM Inference for Distributed Multimodal Mobile Sensing via Progressive Data Orchestration}},
  year = {2026},
  eprint = {2611.20464},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.20464},
}
Figure