Distribution-Consistent Modal Recovering for Incomplete Multimodal Learning

Yuanzhi Wang, Zhen Cui, Yong Li

Introduction

In an ideal scenario, multimodal learning models benefit from a rich stream of synchronized data, such as video, audio, and text. However, real-world applications often face the challenge of Incomplete Multimodal Learning. Sensors fail, background noise obscures audio, and privacy constraints may block camera access, leaving models to operate with missing data streams.

Existing solutions typically attempt to "hallucinate" missing features using encoder-decoder architectures. While these methods can estimate feature values, they often suffer from a significant distribution gap: the recovered data fails to match the statistical properties of the original ground truth. This inconsistency introduces noise that degrades the performance of downstream tasks like sentiment analysis and emotion recognition.

The DiCMoR Framework

To address these limitations, the authors propose DiCMoR (Distribution-Consistent Modal Recovering), a paradigm shift from direct feature estimation to distribution transfer. The framework ensures that recovered data not only looks plausible but also behaves statistically like the original modality.

Detailed architecture of DiCMoR showing the Cross-Modal Distribution Transfer (CMDT) module using Normalizing Flows.

Methodology

DiCMoR operates in three primary stages:

  1. Shallow Feature Extraction: 1D convolutional layers project Language, Visual, and Acoustic modalities into a shared dimensional space.
  2. Cross-Modal Distribution Transfer (CMDT): This core component utilizes Normalizing Flows—generative models that learn invertible mappings between complex data and simple Gaussian distributions. By implementing class-specific flows, the model conditions the distribution on sample labels (e.g., sentiment categories). This prevents "mode collapse," ensuring that the latent space remains discriminative and that "happy" samples do not overlap with "sad" ones.
  3. Refinement and Fusion: A lightweight decoder polishes the recovered features before they are passed, alongside available modalities, into a Multimodal Transformer for final prediction.

By leveraging the invertibility of flows, DiCMoR can take the latent state of an available modality (e.g., text) and reverse it through the flow of a missing modality (e.g., audio) to generate a statistically consistent recovery.

Key Insights and Qualitative Results

The fundamental breakthrough of DiCMoR is the realization that matching the shape of the data distribution is more critical for model robustness than predicting exact feature values.

Comparison of the proposed paradigm (b) against previous methods (a), with t-SNE visualizations (c) showing distribution alignment.

As shown in the t-SNE visualizations above, DiCMoR’s recovered data clusters almost identically to the ground truth. In contrast, previous paradigms often result in scattered or misaligned distributions that confuse downstream classifiers.

Experimental Evaluation

The researchers evaluated DiCMoR on two gold-standard benchmarks for multimodal sentiment analysis: CMU-MOSI and CMU-MOSEI.

  • State-of-the-Art Performance: DiCMoR outperformed existing methods (such as MCTN, MMIN, and GCNet) across nearly all missing modality patterns. On CMU-MOSI, it achieved a binary accuracy of 75.4%, a significant improvement over the previous best of 73.1%.
  • Robustness to Severe Missingness: The model demonstrates "graceful degradation." When the missing data rate was increased from 0.0 to 0.7, DiCMoR’s performance dropped by only 14.6%, whereas competing models saw drops of over 20%.

Conclusion and Future Work

DiCMoR successfully bridges the distribution gap in incomplete multimodal learning by prioritizing statistical consistency. While the current framework requires class labels during training to condition the flows, it sets a new standard for robustness in real-world, sensor-deficient environments. Future research will explore extending this distribution-consistent approach to semi-supervised or unsupervised settings where labels may be partially unavailable.

Summary updated

Abstract

Recovering missing modality is popular in incomplete multimodal learning because it usually benefits downstream tasks. However, the existing methods often directly estimate missing modalities from the observed ones by deep neural networks, lacking consideration of the distribution gap between modalities, resulting in the inconsistency of distributions between the recovered and the true data. To mitigate this issue, in this work, we propose a novel recovery paradigm, Distribution-Consistent Modal Recovering (DiCMoR), to transfer the distributions from available modalities to missing modalities, which thus maintains the distribution consistency of recovered data. In particular, we design a class-specific flow based modality recovery method to transform cross-modal distributions on the condition of sample class, which could well predict a distribution-consistent space for missing modality by virtue of the invertibility and exact density estimation of normalizing flow. The generated data from the predicted distribution is integrated with available modalities for the task of classification. Experiments show that DiCMoR gains superior performances and is more robust than existing state-of-the-art methods under various missing patterns. Visualization results show that the distribution gaps between recovered modalities and missing modalities are mitigated. Codes are released at https://github.com/mdswyz/DiCMoR.
BibTeX
@inproceedings{wang2023distribution,
  author = {Yuanzhi Wang and Zhen Cui and Yong Li},
  title = {{Distribution-Consistent Modal Recovering for Incomplete Multimodal Learning}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2023},
  doi = {10.1109/iccv51070.2023.02013},
}
Figure