Multi-Modal Learning with Missing Modality via Shared-Specific Feature Modelling

Hu Wang, Yuanhong Chen, Congbo Ma, Jodie Avery, Louise Hull, Gustavo Carneiro

TL;DR

This paper introduces ShaSpec, a novel multi-modal learning framework that effectively handles missing modalities during both training and inference. By learning shared and modality-specific features through auxiliary tasks like distribution alignment and domain classification, ShaSpec achieves state-of-the-art performance on both medical image segmentation and computer vision classification tasks, demonstrating remarkable robustness and adaptability.

What Problem Does This Paper Address?

Multi-modal learning, which combines information from various data sources (like images, audio, or text), holds immense promise for improving accuracy in tasks ranging from medical diagnosis to autonomous driving. However, a significant real-world challenge is the frequent absence of one or more modalities, both during training and, more critically, during inference. Existing methods often struggle with this, either by requiring complete data for training, training separate models for each missing modality scenario, or being task-specific (e.g., only for classification or segmentation). This limits their practical applicability and generalizability. The paper aims to bridge this gap by proposing a unified, simpler, and more effective approach that can handle missing modalities gracefully across different tasks.

Methods

The core of the proposed method is Shared-Specific Feature Modelling (ShaSpec). It operates by:

  • Shared and Specific Encoders: For each input modality, ShaSpec employs an individual "specific encoder" to extract modality-unique features and a "shared encoder" to learn features common across all modalities.
  • Residual Feature Fusion: These shared and specific features are then fused using a residual connection. This involves projecting the combined features and adding them back to the original shared features, creating a rich, semantically informative modality embedding.
  • Auxiliary Tasks for Robustness: To enhance the learning of these features, ShaSpec incorporates two auxiliary tasks:
  • Domain Classification Objective (DCO): This encourages specific features to contain modality-specific information by training a classifier to identify the modality of origin.
  • Distribution Alignment Objective (DAO): This aims to make the shared features modality-agnostic by confusing a classifier that tries to distinguish modalities based on shared features. The paper explores different DAO implementations, including cross-entropy loss, KL divergence, and L1/Lp norm distance.
  • Missing Modality Handling: During training and evaluation, if a modality is missing, ShaSpec generates its fused features by leveraging the shared features from the available modalities. This is achieved through a simple generation process, allowing the model to adapt without requiring dedicated training for each missing scenario.
  • Task Adaptability: The architecture is designed to be easily adapted to different tasks. For segmentation, a decoder is used, while for classification, fully connected layers are employed.

Datasets

The paper evaluates ShaSpec on two distinct datasets:

  • BraTS2018: A benchmark for 3D multi-modal brain tumor segmentation, utilizing four MRI modalities (Flair, T1, T1c, T2). This dataset is used for evaluating segmentation performance under missing modality conditions.
  • Audiovision-MNIST: A multi-modal dataset for computer vision classification, combining images (from MNIST) and audio data. This dataset is used to test the model's classification capabilities with varying degrees of missing audio or image modalities.

Key Insights and Ideas

  • The Power of Shared-Specific Features: The paper strongly advocates for explicitly learning both shared (modality-invariant) and specific (modality-unique) features. This dual representation is key to robustly handling missing data, as shared features can compensate for missing modalities, while specific features retain crucial modality-specific information.
  • Auxiliary Tasks as Regularizers: Using distribution alignment and domain classification as auxiliary tasks is a clever way to guide the learning of shared and specific features, respectively. This is more effective than arbitrary orthogonality constraints previously used.
  • Simplicity Enables Adaptability: The straightforward design of ShaSpec, particularly its residual fusion and auxiliary task integration, makes it remarkably easy to adapt to different tasks (segmentation vs. classification) and handle various missing modality scenarios (dedicated vs. non-dedicated training).
  • Missing Modality During Training is Crucial: The paper emphasizes that handling missing modalities during training, not just evaluation, is essential for building truly robust models.

Key Results

  • State-of-the-Art Performance: ShaSpec consistently outperforms existing state-of-the-art methods on both BraTS2018 segmentation and Audiovision-MNIST classification tasks.
  • Significant Improvements on BraTS2018: For non-dedicated training on BraTS2018, ShaSpec achieved improvements of over 3% for enhancing tumor, 5% for tumor core, and 3% for whole tumor Dice scores compared to competing methods. The "ShaSpec*" variant, with prediction smoothness enhancement, further boosted performance.
  • Robust Classification: On Audiovision-MNIST, ShaSpec achieved high classification accuracy (e.g., 93.33% with only 5% audio data), outperforming other models, especially under extreme missing modality conditions.
  • Computational Efficiency: ShaSpec is more computationally efficient than some competing methods, with fewer parameters and faster training/inference times, while also consuming less GPU memory.
  • Feature Visualization: t-SNE visualizations confirm that ShaSpec successfully learns well-separated specific features and clustered shared features, validating the model's design.

Limitations and Future Work

  • Parameter Tuning: While the paper presents sensitivity analysis, finding optimal trade-off weights ($\alpha$ and $\beta$) for the auxiliary losses might still require some tuning.
  • Future Work: The authors suggest extending ShaSpec to other tasks like regression and testing it on a wider range of datasets to further validate its generality and effectiveness. They also hint at exploring other auxiliary tasks or fusion strategies.

Summary updated

Abstract

The missing modality issue is critical but non-trivial to be solved by multi-modal models. Current methods aiming to handle the missing modality problem in multi-modal tasks, either deal with missing modalities only during evaluation or train separate models to handle specific missing modality settings. In addition, these models are designed for specific tasks, so for example, classification models are not easily adapted to segmentation tasks and vice versa. In this paper, we propose the Shared-Specific Feature Modelling (ShaSpec) method that is considerably simpler and more effective than competing approaches that address the issues above. ShaSpec is designed to take advantage of all available input modalities during training and evaluation by learning shared and specific features to better represent the input data. This is achieved from a strategy that relies on auxiliary tasks based on distribution alignment and domain classification, in addition to a residual feature fusion procedure. Also, the design simplicity of ShaSpec enables its easy adaptation to multiple tasks, such as classification and segmentation. Experiments are conducted on both medical image segmentation and computer vision classification, with results indicating that ShaSpec outperforms competing methods by a large margin. For instance, on BraTS2018, ShaSpec improves the SOTA by more than 3% for enhancing tumour, 5% for tumour core and 3% for whole tumour. <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup> <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup> This work received funding from the Australian Government the through Medical Research Futures Fund: Primary Health Care Research Data Infrastructure Grant 2020 and from Endometriosis Australia. G.C. was supported by Australian Research Council through grant FT190100525.
BibTeX
@inproceedings{wang2023multi,
  author = {Hu Wang and Yuanhong Chen and Congbo Ma and Jodie Avery and Louise Hull and Gustavo Carneiro},
  title = {{Multi-Modal Learning with Missing Modality via Shared-Specific Feature Modelling}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2023},
  url = {https://openreview.net/forum?id=PN4I2ldGoDi},
}
Figure