Addressing Missing and Noisy Modalities in One Solution: Unified Modality-Quality Framework for Low-quality Multimodal Data

Sijie Mai, Shiqin Han, Haifeng Hu

TL;DR

The Unified Modality-Quality (UMQ) framework is a new approach to multimodal learning that treats missing and noisy data as a single, unified "low-quality" problem. By combining a rank-guided quality estimator with a specialized Mixture-of-Experts (MoE) architecture, it achieves state-of-the-art results in sentiment and emotion recognition, even when up to 70% of the data is corrupted or absent.

What Problem Does This Paper Address?

In a perfect world, multimodal models (which process text, audio, and video) receive clean, synchronized signals. In the real world, sensors fail, microphones pick up background noise, and cameras get obstructed.

Most existing research treats missing modalities (e.g., no video) and noisy modalities (e.g., static in the audio) as two separate problems requiring different mathematical fixes. However, in practice, these issues often happen simultaneously. This paper argues that we should stop treating them as distinct bugs and instead view them as a unified spectrum of "modality quality." The challenge is building a model that can accurately judge the quality of its own inputs and adapt its processing strategy on the fly without relying on hard-to-get "quality labels."

Methods

The UMQ framework consists of three synergistic components:

  1. Rank-Guided Quality Estimator: Instead of trying to assign an arbitrary "score" (like 0.7 quality) to a video clip, the model uses a ranking strategy. It compares different versions of the same data (e.g., original vs. artificially corrupted) and learns to rank the cleaner version higher. This avoids the "label noise" that comes from trying to define absolute quality.
  2. Quality Enhancer: When a modality is flagged as low-quality, this module attempts to "repair" it. It does this by leveraging two things:
  3. Sample-specific info: Context borrowed from the other healthy modalities.
  4. Modality-specific info: A "modality baseline" that represents the global characteristics of that medium (e.g., what "typical" acoustic features look like), ensuring the reconstructed data still feels like audio or video.
  5. Modality-Quality-Aware Mixture-of-Experts (MQ-MoE): Since there are many combinations of "good" and "bad" modalities (e.g., good text + bad audio vs. bad text + missing video), a single predictor isn't enough. UMQ uses a "router" to send the data to specialized "expert" sub-networks trained to handle specific quality configurations.

Datasets

The authors evaluated UMQ on five widely-used Multimodal Affective Computing (MAC) benchmarks:

  • CMU-MOSI & CMU-MOSEI: The gold standards for multimodal sentiment analysis.
  • CH-SIMS: A Chinese multimodal sentiment dataset with fine-grained labels.
  • UR-FUNNY: A dataset for multimodal humor detection.
  • MUStARD: A dataset for multimodal sarcasm detection.

Key Insights and Ideas

  • Noise is just "Extreme" Low Quality: By treating a missing modality as a known pattern of extreme noise, the authors simplify the architecture. This allows one model to handle everything from a slightly blurry video to a completely dead camera feed.
  • The Power of Ranking: One of the biggest hurdles in quality estimation is that we don't have ground-truth "quality scores." By using a ranking loss—essentially telling the model "this version is better than that version"—the estimator becomes much more robust and easier to train.
  • Avoiding "Identity Loss": A common failure in data reconstruction is that the model creates a "hallucinated" version of the missing data that loses its unique modality characteristics. UMQ’s use of a modality baseline ensures that reconstructed audio still "sounds" like audio, preserving the inherent properties of the medium.

Key Results

UMQ consistently outperformed State-of-the-Art (SOTA) baselines across all datasets and settings:

  • Complete Data: Even when data was perfect, UMQ set new records (e.g., 90.1% binary accuracy on CMU-MOSI), proving that quality awareness helps even with "clean" data.
  • Missing Data: At a 70% missing rate (Missing Rate = 0.7), UMQ maintained significantly higher accuracy than specialized models like GCNet and MMIN. On CMU-MOSEI, it achieved 80.2% accuracy averaged across all missingness levels.
  • Noisy Data: When subjected to varying levels of Gaussian noise, UMQ's performance remained stable, while other models saw sharp declines. It also generalized well to "unseen" noise types like Laplace noise and random erasing.

Limitations and Future Work

While UMQ is powerful, the authors note a few areas for growth:

  • Computational Complexity: The Mixture-of-Experts (MoE) approach requires managing multiple sub-networks. While only a few are active at a time (via the router), the total parameter count is higher than simpler fusion models.
  • Memory Constraints: Updating the "modality baseline" representation requires a moving average of features, which can be sensitive to the batch size and hardware memory limits.
  • Future Direction: The authors suggest exploring how this unified quality framework could be applied to even larger Multimodal Large Language Models (MLLMs), which currently struggle with "hallucinations" when faced with low-quality visual or audio inputs.

Summary updated

Abstract

Multimodal data encountered in real-world scenarios are typically of low quality, with noisy modalities and missing modalities being typical forms that severely hinder model performance and robustness. However, prior works often handle noisy and missing modalities separately. In contrast, we jointly address missing and noisy modalities to enhance model robustness in low-quality data scenarios. We regard both noisy and missing modalities as a unified low-quality modality problem, and propose a unified modality-quality (UMQ) framework to enhance low-quality representations for multimodal affective computing. Firstly, we train a quality estimator with explicit supervised signals via a rank-guided training strategy that compares the relative quality of different representations by adding a ranking constraint, avoiding training noise caused by inaccurate absolute quality labels. Then, a quality enhancer for each modality is constructed, which uses the sample-specific information provided by other modalities and the modality-specific information provided by the defined modality baseline representation to enhance the quality of unimodal representations. Finally, we propose a quality-aware mixture-of-experts module with particular routing mechanism to enable multiple modality-quality problems to be addressed more specifically. UMQ consistently outperforms state-of-the-art baselines on multiple datasets under the settings of complete, missing, and noisy modalities.
BibTeX
@misc{mai2026addressing,
  author = {Sijie Mai and Shiqin Han and Haifeng Hu},
  title = {{Addressing Missing and Noisy Modalities in One Solution: Unified Modality-Quality Framework for Low-quality Multimodal Data}},
  year = {2026},
  eprint = {2603.02695},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2603.02695},
}
Figure