TL;DR
Pretrained multimodal networks suffer when modalities are missing at inference, and existing imputation methods fail to assess how trustworthy their reconstructed inputs actually are. SURE (Scalable Uncertainty and Reconstruction Estimation) solves this by freezing the pretrained model, learning lightweight latent reconstructors, and tracking uncertainty from missing inputs through to the final downstream prediction using a distribution-free Pearson correlation loss and classical error propagation.
What Problem Does This Paper Address?
In real-world applications like medical diagnosis or autonomous driving, sensor failures and missing inputs are common. While multimodal foundation models achieve state-of-the-art results when all inputs are present, they struggle on small-scale datasets with missing modalities.
Prior work typically trains generative modules (like VAEs or masked autoencoders) to reconstruct missing features. However, these reconstructions are treated as ground truth—even when the imputation is inaccurate. Standard models lack the ability to quantify their own confidence and say "I don't know" when making predictions from low-quality reconstructions.
Methods
SURE is a plug-and-play framework designed to sit on top of any pretrained multimodal backbone with minimal retraining:
[Available Modalities] ---> [Pretrained Projectors (Frozen)]
|
v
[Latent Reconstruction Module r(·) (Trainable)]
|
+-------------------+-------------------+
| |
Reconstructed Latents Reconstruction Uncertainty (σ²)
| |
v v
[Fusion Backbone (Frozen)] -------------> [Error Propagation (∂ω/∂z)²]
| |
v v
[Downstream Head] ---------------------> [Total Output Uncertainty]
Key technical elements include:
- Latent-Space Reconstruction: Rather than raw data imputation, lightweight multi-layer perceptrons ($r^i(\cdot)$) reconstruct missing modality representations directly within the shared latent space between unimodal projectors and the fusion backbone.
-
Distribution-Free Uncertainty Loss ($L_{\text{PCC}}$): Standard Bayesian uncertainty estimation relies on Gaussian Negative Log-Likelihood (NLL). When prediction error approaches zero ($\tilde{\epsilon}^2 \to 0$), the Gaussian NLL gradient collapses to an undefined $0/0$ form, causing optimization instability. SURE replaces NLL with a Pearson Correlation Coefficient loss: $$\mathcal{L}_{\text{PCC}}(\tilde{\sigma}^2, \tilde{\epsilon}^2) = 1 - r(\tilde{\sigma}^2, \tilde{\epsilon}^2)$$ This guarantees that predicted variance $\tilde{\sigma}^2$ scales monotonically with actual squared error $\tilde{\epsilon}^2$ without imposing strict distributional shapes or arbitrary variance bounds.
-
Statistical Error Propagation in Deep Networks: To capture how reconstruction noise impacts final predictions, the authors adopt the classical physical measurement formula for variance propagation: $$\tilde{\sigma}^2_{\text{input}} \approx \sum_{i \in \mathcal{J}} \left(\frac{\partial \omega}{\partial \tilde{Z}^i}\right)^2 \tilde{\sigma}^2_{\tilde{Z}^i}$$ where $\omega$ is the frozen fusion network and $\mathcal{J}$ indexes missing modalities.
-
Unified Output Variance: Total prediction uncertainty combines propagated input uncertainty and the model's inherent aleatoric uncertainty ($\tilde{\sigma}^2_Y = \tilde{\sigma}^2_{\text{input}} + \tilde{\sigma}^2_\omega$).
Datasets
SURE was evaluated by fine-tuning pretrained architectures across three distinct multimodal tasks, artificially masking out 50% of each modality during training and testing:
- Sentiment Analysis: Fine-tuned on incomplete CMU-MOSI (text, video, audio) using weights pretrained on CMU-MOSEI.
- Book Genre Classification: Fine-tuned on the Book Dataset (text summaries + cover images) using weights pretrained on movie genre classification (MM-IMDB).
- Human Action Recognition: Fine-tuned on incomplete UTD-MHAD (RGB video, smartwatch acceleration, phone gyroscope) using weights pretrained on MMAct.
Key Insights and Ideas
- Correlation Beats Likelihood in Practice: Enforcing a high linear correlation between error and uncertainty via $L_{\text{PCC}}$ yields smoother convergence and avoids the exploding/vanishing gradients typical of heteroscedastic Gaussian NLL near zero error.
- Uncertainty Chains Across Layers: Downstream prediction error is strongly driven by reconstruction quality. Propagating intermediate latent uncertainties through the frozen fusion layers via network Jacobians accurately predicts which downstream outputs are likely to fail.
- Actionable Selective Deferral: In decision-making simulations, setting uncertainty thresholds allowed the system to defer high-risk predictions to human reviewers, significantly improving accuracy on the retained set while optimizing the True Deferral Rate over unnecessary deferrals.
Key Results
- Downstream Accuracy: On CMU-MOSI, SURE achieved a full-modality MAE of 0.583 and F1 of 0.891, dramatically outperforming reconstruction baselines like ActionMAE (MAE 1.005, F1 0.719) and DiCMoR (MAE 1.106, F1 0.650), while also surpassing prompt-tuning methods (MPMM, MPLMM).
- Uncertainty Calibration: On UTD-MHAD action recognition, SURE produced a reconstruction uncertainty correlation of 0.837–0.878 (compared to 0.115–0.166 for Gaussian MLE) and reduced output Uncertainty Calibration Error (UCE) to 0.009 on full inputs (vs. 0.281 for Gaussian MLE).
- Ablations: Removing either the latent reconstructors or the error propagation logic caused notable drops in downstream F1 scores (e.g., dropping from 0.739 to 0.151 when incomplete samples were ignored outright).
Limitations and Future Work
- Modality Imbalance: Certain information-rich modalities (e.g., text in sentiment analysis or accelerometer in action recognition) disproportionately dominate the latent space, making missingness in these modalities much harder to recover.
- Independence Assumptions: The error propagation formula assumes latent dimension uncertainties are uncorrelated, which may overlook cross-feature covariance.
- Overestimation Bias: Test visualizations showed a slight tendency to overestimate both reconstruction and output uncertainties on outlier samples.
Summary updated