DisCoM-KD: Cross-Modal Knowledge Distillation via Disentanglement Representation and Adversarial Learning

Dino Ienco, Cássio Fraga Dantas

Abstract

Cross-modal knowledge distillation (CMKD) is a critical technique for deploying robust models in environments where only a single modality is available at inference time, despite the availability of multi-modal data during training. Traditional CMKD methods typically rely on a teacher-student paradigm, which introduces computational overhead and often fails to capture modality-discriminative features. This paper presents DisCoM-KD, a novel framework that replaces the teacher-student setup with a joint training approach based on feature disentanglement and adversarial learning. By separating features into modality-invariant, modality-informative, and modality-irrelevant components, DisCoM-KD enables single-modal classifiers to inherit the "dark knowledge" of multi-modal contexts. Experimental results across three benchmarks demonstrate that DisCoM-KD consistently outperforms state-of-the-art distillation methods.

Introduction

In modern machine learning, multi-modal datasets—combining sources such as RGB images, depth maps, thermal data, and SAR imagery—provide a rich representation of the world. However, a significant gap exists between training and deployment: while we may have access to multiple sensors during the research phase, the deployment hardware is often limited to a single modality due to cost, power, or environmental constraints. This challenge, known as modality mismatch, necessitates effective Cross-Modal Knowledge Distillation (CMKD).

Standard CMKD approaches generally employ a pre-trained multi-modal "teacher" to guide a single-modal "student." While functional, this paradigm suffers from several limitations: the arbitrary selection and weighting of teacher modalities, the high computational cost of sequential training, and a tendency to lose modality-specific nuances that are crucial for high-accuracy classification. DisCoM-KD (Disentanglement-learning based Cross-Modal Knowledge Distillation) addresses these issues by rethinking the distillation process as a simultaneous, disentangled learning task.

Methodology

DisCoM-KD moves away from the sequential teacher-student model in favor of two parallel branches (one for each modality) that are trained jointly but remain capable of independent deployment. The framework's efficacy relies on its ability to disentangle the latent feature space into three distinct types of embeddings for every input:

  1. Modality-Invariant ($z^{inv}$): These features represent information shared across modalities, such as the structural geometry of an object visible in both RGB and Depth data.
  2. Modality-Informative ($z^{inf}$): These are features unique to a specific modality that contribute positively to the task, such as the specific texture in an RGB image or the heat signature in a thermal map.
  3. Modality-Irrelevant ($z^{irr}$): These represent noise or modality-specific artifacts that do not contribute to the final classification task.

Disentanglement and Adversarial Learning

To achieve this separation, the authors employ a sophisticated combination of loss functions and architectural components:

  • Adversarial Training with GRL: A Gradient Reversal Layer (GRL) is integrated with a modality classifier. The system penalizes the feature extractor if the classifier can successfully identify the source modality of $z^{inv}$, effectively forcing $z^{inv}$ to become modality-agnostic.
  • Orthogonality Constraints: To ensure that the three embedding types do not contain redundant information, an orthogonality loss is applied, forcing the vectors to be mathematically perpendicular to one another.
  • Auxiliary Classifiers: To maintain the discriminative power of the features, intermediate classifiers are applied to both the invariant and informative embeddings, ensuring that the disentanglement process does not discard task-relevant information.

Experimental Evaluation

The framework was rigorously tested on three multi-modal benchmarks representing different domains:

  • SUNRGBD: Scene classification using RGB and Depth images.
  • EuroSat-MS-SAR: Land cover classification using Multi-Spectral and Synthetic Aperture Radar (SAR) satellite imagery.
  • TRISTAR: Action recognition using Thermal and Depth sensors.

Performance and Key Results

DisCoM-KD demonstrated superior performance compared to state-of-the-art (SOTA) methods such as Decoupled Knowledge Distillation (DKD) and Multi-Level Knowledge Distillation (MLKD).

On the SUNRGBD dataset (distilling knowledge from RGB to Depth), DisCoM-KD achieved an F1-Score of 47.69, a significant improvement over the leading competitor's score of 42.87. Notably, while many existing CMKD methods occasionally performed worse than a simple baseline student, DisCoM-KD was the only framework to provide consistent performance gains across all tested scenarios. In "all-to-one" distillation tasks on the EuroSat dataset, the framework reached an F1-Score of 98.12, occasionally surpassing the performance of the multi-modal teacher itself.

Discussion

The success of DisCoM-KD highlights several key insights for the field of multi-modal learning:

  • The Teacher-less Advantage: The results suggest that a pre-trained teacher is not strictly necessary for effective distillation. Joint training with explicit disentanglement allows single-modal branches to regularize each other, capturing the benefits of multi-modal context naturally.
  • Robustness through Disentanglement: By explicitly separating shared information from modality-specific information, the model becomes more robust to the noise inherent in individual sensors.
  • Efficiency: Training all classifiers simultaneously reduces the multi-stage overhead associated with traditional KD, though the complexity of the training loop itself increases due to the multiple loss terms.

Conclusion and Future Work

DisCoM-KD represents a significant step forward in cross-modal knowledge distillation, offering a more principled approach to handling modality mismatch than traditional teacher-student paradigms. While the current framework is optimized for pairs of modalities and classification tasks, future research will explore its scalability to $N$-modality systems and its applicability to dense prediction tasks such as semantic segmentation and object detection. Despite the increased complexity of the training objectives, the consistent performance gains and the elimination of the pre-trained teacher make DisCoM-KD a compelling choice for real-world multi-modal deployment.

Summary updated

BibTeX
@inproceedings{ienco2025discom,
  author = {Dino Ienco and Cássio Fraga Dantas},
  title = {{DisCoM-KD: Cross-Modal Knowledge Distillation via Disentanglement Representation and Adversarial Learning}},
  booktitle = {BMVC},
  year = {2025},
  url = {https://openreview.net/forum?id=YF7Ausjjj6},
}
Figure