Are you SURE? Enhancing Multimodal Pretraining with Missing Modalities through Uncertainty Estimation

Duy A. Nguyen, Quan Huu Do, Khoa D. Doan, Minh N. Do

Abstract

Multimodal learning has demonstrated incredible successes by integrating diverse data sources, yet it often relies on the availability of all modalities - an assumption that rarely holds in real-world applications. Pretrained multimodal models, while effective, struggle when confronted with small-scale and incomplete datasets (i.e., missing modalities), limiting their practical applicability. Previous studies on reconstructing missing modalities have overlooked the reconstruction's potential unreliability, which could compromise the quality of the final outputs. We present SURE (Scalable Uncertainty and Reconstruction Estimation), a novel framework that extends the capabilities of pretrained multimodal models by introducing latent space reconstruction and uncertainty estimation for both reconstructed modalities and downstream tasks. Our method is architecture-agnostic, reconstructs missing modalities, and delivers reliable uncertainty estimates, improving both interpretability and performance. SURE introduces a unique Pearson Correlation-based loss and applies statistical error propagation in deep networks for the first time, allowing precise quantification of uncertainties from missing data and model predictions. Extensive experiments across tasks such as sentiment analysis, genre classification, and action recognition show that SURE consistently achieves state-of-the-art performance, ensuring robust predictions even in the presence of incomplete data.

Introduction

Multimodal AI models often crumble when data is missing—for instance, when a camera fails or a microphone cuts out—yet they rarely admit when they are guessing. While multimodal learning has demonstrated incredible successes by integrating diverse data sources, it often relies on the availability of all modalities, an assumption that rarely holds in real-world applications like autonomous vehicles or medical diagnostics. Previous studies on reconstructing missing modalities have overlooked the potential unreliability of these reconstructions. We present SURE (Scalable Uncertainty and Reconstruction Estimation), a framework that reconstructs missing data in the latent space and, for the first time, uses statistical error propagation to quantify exactly how much to trust the final prediction.

Methodology

The SURE framework introduces a three-pronged approach to handle missing modalities reliably:

  1. Latent Space Reconstruction: Instead of computationally expensive raw data generation, SURE inserts small "reconstructor" modules after initial feature extraction. These modules learn to approximate missing features of one modality using available features from others.
  2. Pearson Correlation-Based (PCC) Loss: Moving away from standard Negative Log-Likelihood (NLL) loss, SURE uses a distribution-free loss based on the Pearson Correlation Coefficient. This ensures the model’s uncertainty score correlates linearly with actual prediction error without strict magnitude constraints.
  3. Deep Error Propagation: SURE adapts classical statistical error propagation (the "delta method") to track uncertainty through the network. Shakiness in reconstruction is mathematically propagated through frozen fusion layers to the final output.
The SURE framework architecture showing latent space reconstruction and error propagation.

Experimental Setup

SURE was evaluated across three distinct tasks to demonstrate versatility:

  • Sentiment Analysis: CMU-MOSI dataset (video, audio, text).
  • Book Genre Classification: Titles, summaries, and cover images.
  • Human Action Recognition: UTD-MHAD dataset (video, accelerometer, gyroscope). The framework was integrated with models pre-trained on larger datasets like CMU-MOSEI and MMAct to demonstrate compatibility with high-performance backbones.

Results and Discussion

SURE consistently achieved state-of-the-art (SOTA) results across all tested benchmarks. In sentiment analysis on CMU-MOSI, SURE outperformed specialized baselines like IMDer and DiCMoR in F1-score and Accuracy.

A key insight is the interdependence of modalities: higher reconstruction uncertainty almost always leads to higher output uncertainty. This confirms that the quality of hallucinated data directly impacts final decision reliability.

Relationship between reconstruction uncertainty/error and output uncertainty/error across different modality configurations.

Furthermore, SURE's uncertainty estimates are significantly more reliable than standard methods. For example, in the UTD-MHAD dataset, SURE achieved a "Reconstruct Uncertainty Correlation" of 0.878, compared to just 0.166 for Gaussian MLE baselines. This reliability allows uncertainty to act as a "safety valve," where models can defer low-confidence decisions to human experts.

Conclusion

While SURE provides robust predictions even with 50% missing data, it faces challenges with Modality Imbalance, where a dominant modality's absence is hard to overcome. Future work will focus on multi-modality missingness and refining uncertainty estimates to avoid occasional overestimation.

Summary updated

Abstract

Multimodal learning has demonstrated incredible successes by integrating diverse data sources, yet it often relies on the availability of all modalities - an assumption that rarely holds in real-world applications. Pretrained multimodal models, while effective, struggle when confronted with small-scale and incomplete datasets (i.e., missing modalities), limiting their practical applicability. Previous studies on reconstructing missing modalities have overlooked the reconstruction's potential unreliability, which could compromise the quality of the final outputs. We present SURE (Scalable Uncertainty and Reconstruction Estimation), a novel framework that extends the capabilities of pretrained multimodal models by introducing latent space reconstruction and uncertainty estimation for both reconstructed modalities and downstream tasks. Our method is architecture-agnostic, reconstructs missing modalities, and delivers reliable uncertainty estimates, improving both interpretability and performance. SURE introduces a unique Pearson Correlation-based loss and applies statistical error propagation in deep networks for the first time, allowing precise quantification of uncertainties from missing data and model predictions. Extensive experiments across tasks such as sentiment analysis, genre classification, and action recognition show that SURE consistently achieves state-of-the-art performance, ensuring robust predictions even in the presence of incomplete data.
BibTeX
@misc{nguyen2025are,
  author = {Duy A. Nguyen and Quan Huu Do and Khoa D. Doan and Minh N. Do},
  title = {{Are you SURE? Enhancing Multimodal Pretraining with Missing Modalities through Uncertainty Estimation}},
  year = {2025},
  eprint = {2504.13465},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2504.13465},
}
Figure