Are you SURE? Enhancing Multimodal Pretraining with Missing Modalities through Uncertainty Estimation

Duy A. Nguyen, Do Huu Quan, Khoa D Doan, Minh N. DoSubmission to ICLR 2025 · Withdrawn

TL;DR

Pretrained multimodal networks suffer when modalities are missing at inference, and existing imputation methods fail to assess how trustworthy their reconstructed inputs actually are. SURE (Scalable Uncertainty and Reconstruction Estimation) solves this by freezing the pretrained model, learning lightweight latent reconstructors, and tracking uncertainty from missing inputs through to the final downstream prediction using a distribution-free Pearson correlation loss and classical error propagation.

What Problem Does This Paper Address?

In real-world applications like medical diagnosis or autonomous driving, sensor failures and missing inputs are common. While multimodal foundation models achieve state-of-the-art results when all inputs are present, they struggle on small-scale datasets with missing modalities.

Prior work typically trains generative modules (like VAEs or masked autoencoders) to reconstruct missing features. However, these reconstructions are treated as ground truth—even when the imputation is inaccurate. Standard models lack the ability to quantify their own confidence and say "I don't know" when making predictions from low-quality reconstructions.

Methods

SURE is a plug-and-play framework designed to sit on top of any pretrained multimodal backbone with minimal retraining:

[Available Modalities] ---> [Pretrained Projectors (Frozen)] 
                                   |
                                   v
             [Latent Reconstruction Module r(·) (Trainable)]
                                   |
               +-------------------+-------------------+
               |                                       |
    Reconstructed Latents                   Reconstruction Uncertainty (σ²)
               |                                       |
               v                                       v
    [Fusion Backbone (Frozen)] -------------> [Error Propagation (∂ω/∂z)²]
               |                                       |
               v                                       v
      [Downstream Head] ---------------------> [Total Output Uncertainty]

Key technical elements include:

  • Latent-Space Reconstruction: Rather than raw data imputation, lightweight multi-layer perceptrons ($r^i(\cdot)$) reconstruct missing modality representations directly within the shared latent space between unimodal projectors and the fusion backbone.
  • Distribution-Free Uncertainty Loss ($L_{\text{PCC}}$): Standard Bayesian uncertainty estimation relies on Gaussian Negative Log-Likelihood (NLL). When prediction error approaches zero ($\tilde{\epsilon}^2 \to 0$), the Gaussian NLL gradient collapses to an undefined $0/0$ form, causing optimization instability. SURE replaces NLL with a Pearson Correlation Coefficient loss: $$\mathcal{L}_{\text{PCC}}(\tilde{\sigma}^2, \tilde{\epsilon}^2) = 1 - r(\tilde{\sigma}^2, \tilde{\epsilon}^2)$$ This guarantees that predicted variance $\tilde{\sigma}^2$ scales monotonically with actual squared error $\tilde{\epsilon}^2$ without imposing strict distributional shapes or arbitrary variance bounds.

  • Statistical Error Propagation in Deep Networks: To capture how reconstruction noise impacts final predictions, the authors adopt the classical physical measurement formula for variance propagation: $$\tilde{\sigma}^2_{\text{input}} \approx \sum_{i \in \mathcal{J}} \left(\frac{\partial \omega}{\partial \tilde{Z}^i}\right)^2 \tilde{\sigma}^2_{\tilde{Z}^i}$$ where $\omega$ is the frozen fusion network and $\mathcal{J}$ indexes missing modalities.

  • Unified Output Variance: Total prediction uncertainty combines propagated input uncertainty and the model's inherent aleatoric uncertainty ($\tilde{\sigma}^2_Y = \tilde{\sigma}^2_{\text{input}} + \tilde{\sigma}^2_\omega$).

Datasets

SURE was evaluated by fine-tuning pretrained architectures across three distinct multimodal tasks, artificially masking out 50% of each modality during training and testing:

  • Sentiment Analysis: Fine-tuned on incomplete CMU-MOSI (text, video, audio) using weights pretrained on CMU-MOSEI.
  • Book Genre Classification: Fine-tuned on the Book Dataset (text summaries + cover images) using weights pretrained on movie genre classification (MM-IMDB).
  • Human Action Recognition: Fine-tuned on incomplete UTD-MHAD (RGB video, smartwatch acceleration, phone gyroscope) using weights pretrained on MMAct.

Key Insights and Ideas

  • Correlation Beats Likelihood in Practice: Enforcing a high linear correlation between error and uncertainty via $L_{\text{PCC}}$ yields smoother convergence and avoids the exploding/vanishing gradients typical of heteroscedastic Gaussian NLL near zero error.
  • Uncertainty Chains Across Layers: Downstream prediction error is strongly driven by reconstruction quality. Propagating intermediate latent uncertainties through the frozen fusion layers via network Jacobians accurately predicts which downstream outputs are likely to fail.
  • Actionable Selective Deferral: In decision-making simulations, setting uncertainty thresholds allowed the system to defer high-risk predictions to human reviewers, significantly improving accuracy on the retained set while optimizing the True Deferral Rate over unnecessary deferrals.

Key Results

  • Downstream Accuracy: On CMU-MOSI, SURE achieved a full-modality MAE of 0.583 and F1 of 0.891, dramatically outperforming reconstruction baselines like ActionMAE (MAE 1.005, F1 0.719) and DiCMoR (MAE 1.106, F1 0.650), while also surpassing prompt-tuning methods (MPMM, MPLMM).
  • Uncertainty Calibration: On UTD-MHAD action recognition, SURE produced a reconstruction uncertainty correlation of 0.837–0.878 (compared to 0.115–0.166 for Gaussian MLE) and reduced output Uncertainty Calibration Error (UCE) to 0.009 on full inputs (vs. 0.281 for Gaussian MLE).
  • Ablations: Removing either the latent reconstructors or the error propagation logic caused notable drops in downstream F1 scores (e.g., dropping from 0.739 to 0.151 when incomplete samples were ignored outright).

Limitations and Future Work

  • Modality Imbalance: Certain information-rich modalities (e.g., text in sentiment analysis or accelerometer in action recognition) disproportionately dominate the latent space, making missingness in these modalities much harder to recover.
  • Independence Assumptions: The error propagation formula assumes latent dimension uncertainties are uncorrelated, which may overlook cross-feature covariance.
  • Overestimation Bias: Test visualizations showed a slight tendency to overestimate both reconstruction and output uncertainties on outlier samples.

Summary updated

Abstract

Multimodal learning has demonstrated incredible successes by integrating diverse data sources, yet it often relies on the availability of all modalities - an assumption that rarely holds in real-world applications. Pretrained multimodal models, while effective, struggle when confronted with small-scale and incomplete datasets (i.e., missing modalities), limiting their practical applicability. Previous studies on reconstructing missing modalities have overlooked the reconstruction's potential unreliability, which could compromise the quality of the final outputs. We present **SURE** (Scalable Uncertainty and Reconstruction Estimation), a novel framework that extends the capabilities of pretrained multimodal models by introducing latent space reconstruction and uncertainty estimation for both reconstructed modalities and downstream tasks. Our method is architecture-agnostic, reconstructs missing modalities, and delivers reliable uncertainty estimates, improving both interpretability and performance. SURE introduces a unique Pearson Correlation-based loss and applies statistical error propagation in deep networks for the first time, allowing precise quantification of uncertainties from missing data and model predictions. Extensive experiments across tasks such as sentiment analysis, genre classification, and action recognition show that SURE consistently achieves state-of-the-art performance, ensuring robust predictions even in the presence of incomplete data.
BibTeX
@misc{nguyen2025are,
  author = {Duy A. Nguyen and Do Huu Quan and Khoa D Doan and Minh N. Do},
  title = {{Are you SURE? Enhancing Multimodal Pretraining with Missing Modalities through Uncertainty Estimation}},
  year = {2025},
  eprint = {2504.13465},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2504.13465},
}

Loading related content...

Figure