MMANet: Margin-Aware Distillation and Modality-Aware Regularization for Incomplete Multimodal Learning

Shicai Wei, Chunbo Luo, Yang LuoCVPR 2023 · DOI · arXiv

Introduction

Multimodal learning, which integrates diverse data types such as RGB images, depth maps, and infrared (IR) scans, has significantly advanced fields like face anti-spoofing and semantic segmentation. By leveraging complementary information from different sensors, these models achieve higher accuracy than unimodal counterparts. However, real-world deployment often faces the challenge of Incomplete Multimodal Learning (IML). Sensors may fail, environmental conditions (like lighting) might render a modality useless, or privacy constraints may restrict access to certain data.

Existing solutions typically involve "unified models"—single networks trained to handle any subset of available inputs. While efficient, these models often underperform compared to "customized models" designed for specific modality combinations. Unified models tend to focus on modality-invariant features, neglecting the unique, modality-specific information that individual sensors provide. Furthermore, they often suffer from an imbalance where the network over-relies on the strongest modality, leading to suboptimal performance on weaker or incomplete input sets.

The MMANet Framework

To address these challenges, the authors propose MMANet, a framework designed to bridge the gap between unified and customized models. MMANet consists of three primary components: a Teacher Network, a Deployment Network, and a Regularization Network.

The overall architecture of MMANet, featuring Teacher, Deployment, and Regularization networks.

The Teacher Network is pre-trained on complete multimodal data to capture fine-grained inter-class relationships. The Deployment Network is the student model intended for production, trained to handle missing modalities. The Regularization Network acts as an adaptive tutor during training.

Margin-aware Distillation (MAD)

Standard Knowledge Distillation (KD) encourages a student to mimic a teacher's output. However, not all samples are equally informative. MMANet introduces Margin-aware Distillation (MAD), which utilizes classification uncertainty to identify "hard" samples near decision boundaries. By calculating the entropy of the teacher’s predictions, MAD identifies samples where the teacher is less certain. These boundary samples contain critical information about the "inter-class margin." By weighting these samples more heavily, MAD forces the deployment network to learn the refined boundaries that are typically only perceivable when all data modalities are present.

Modality-aware Regularization (MAR)

Multimodal models often become "lazy," relying heavily on dominant modalities (like RGB) while neglecting others. Modality-aware Regularization (MAR) is an adaptive strategy that identifies which modality combinations the model is currently struggling with—the "weak" combinations. Using a contrastive ranking strategy and a memory bank to track performance over time, MAR dynamically adjusts the training focus. It forces the network to minimize additional loss for these weak spots, ensuring balanced performance across all possible input combinations.

Key Results and Insights

The researchers evaluated MMANet on two distinct tasks: multimodal classification and semantic segmentation.

Performance Benchmarks

MMANet demonstrated superior performance across all benchmarks:

  • Classification: On the CASIA-SURF dataset (RGB, Depth, IR), MMANet achieved an Average Classification Error Rate (ACER) of 3.94%, a substantial reduction from the previous state-of-the-art unified model, mmFormer (5.93%). Notably, MMANet even outperformed the customized baseline (5.80%).
  • Segmentation: On the NYUv2 dataset, it reached a mean Intersection over Union (mIOU) of 45.58%, outperforming the next best method by 1.32%.
Quantitative results on CASIA-SURF and visualization comparing Similarity-Preserving (SP) distillation with MAD.

Core Insights

  • Boundary Samples are Gold: Not all data points are equally useful for distillation. Samples near the decision boundary contain the most critical information about how to distinguish between classes.
  • Modality Importance is Fluid: A modality that is "strong" in one scenario might be "weak" in another. MAR’s ability to mine weak combinations dynamically is more effective than static weighting.
  • The "Memorization Effect": The authors leverage the observation that deep networks tend to learn simple, "strong" patterns first before overfitting to harder ones. MAR uses this "warm-up" phase to identify which modalities the model finds easiest to rely on, then pivots to focus on the harder ones.

Conclusion

MMANet represents a significant step forward in Incomplete Multimodal Learning, offering a robust framework that maintains high performance even when data is missing. While highly effective, the framework currently depends on a teacher model trained on complete data. Future research could explore adaptations for scenarios with permanently incomplete training data or scaling the framework to handle a larger number of sensors.

Summary updated

BibTeX
@inproceedings{wei2023mmanet,
  author = {Shicai Wei and Chunbo Luo and Yang Luo},
  title = {{MMANet: Margin-Aware Distillation and Modality-Aware Regularization for Incomplete Multimodal Learning}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2023},
  doi = {10.1109/cvpr52729.2023.01919},
}
Figure