TL;DR
TALD (Transferability-Aware Logit Disentanglement) is a new framework that prevents "negative transfer" in cross-modal knowledge distillation by separating modality-general knowledge from modality-specific noise. By working directly in the class-aligned logit space, it achieves superior accuracy gains across vision, audio, and text benchmarks while remaining significantly more efficient than previous feature-based methods.
What Problem Does This Paper Address?
In Cross-Modal Knowledge Distillation (CMKD), we try to teach a student model in one modality (e.g., audio) using a pre-trained teacher from another (e.g., video). The hope is that the teacher's rich representations will help the student learn faster or better.
However, modalities aren't identical. A video teacher might rely on visual cues (like lip movement) that simply don't exist for an audio student. If the teacher forces these "modality-specific" cues onto the student, it can lead to negative transfer—where the student actually performs worse than if it had just trained on its own.
The challenge is: how do we extract only the "modality-general" knowledge (the stuff that applies to both) while ignoring the modality-specific "noise"?
Methods
The authors propose TALD, a two-stage framework that performs disentanglement in the logit space (the final class scores before softmax). Logits are "class-aligned," meaning the $i$-th dimension always represents the $i$-th class, regardless of modality, making them a compact and direct space for modeling transferability.
Stage 1: Transferability-Aware Disentanglement
TALD uses a Residual Disentangler to split logits into two parts:
- Modality-General (GEN): Information useful across modalities.
- Modality-Specific (SPEC): Information unique to the source modality.
This split is guided by two clever criteria:
- C1 (Self-benefiting): The SPEC component should help classification within its own modality.
- C2 (Cross-modal exclusivity): Adding the SPEC component from one modality to the GEN logits of another should increase error (cross-entropy), proving it's truly modality-specific.
To handle the fact that some samples are easier to transfer than others, a Sample-Adaptive Modulator predicts a per-sample "margin." This margin acts as a control signal, relaxing or tightening the disentanglement constraints based on the specific input.
Stage 2: Adaptive Knowledge Distillation
Once the GEN components are learned, TALD creates a "clean" distillation target by adaptively fusing the GEN logits from the teacher and a "reference student" (a model trained only on the target modality). This ensures the final student receives supervision that is both expert-informed and modality-appropriate.
Datasets
The authors evaluated TALD across five diverse multimodal benchmarks:
- AV-MNIST: Image and audio of handwritten digits.
- RAVDESS: Video and audio for emotion recognition.
- VGGSound-50k: Video and audio for scene classification.
- CrisisMMD-V2: Image and text for humanitarian crisis classification.
- Fakeddit: Image and text for fine-grained fake news detection.
Key Insights and Ideas
- Logits are enough: Most prior work tries to filter high-dimensional features, which is computationally expensive. TALD proves that disentangling at the logit level is not only more efficient but also more effective because it directly targets the classification task.
- The "Residual" Trick: By defining the modality-general component as the original logit minus a learned specific component ($z_{gen} = z - z_{spec}$), the model maintains an exact additive relationship, ensuring the disentanglement is grounded in the original predictions.
- Exclusivity as a Filter: The idea that "specific" info should actively hurt a different modality (C2 criterion) is a powerful inductive bias for ensuring the GEN component is truly universal.
Key Results
- Superior Accuracy: TALD achieved the largest mean accuracy gain (+1.38 percentage points) over the "no distillation" baseline across all ten transfer directions.
- Reliability: It showed a 84% positive-transfer frequency, meaning it almost always helped the student, whereas baseline methods often suffered from negative transfer (improving in only 10% to 70% of cases).
- Efficiency: Compared to the state-of-the-art MGDFR, TALD reduced total FLOPs by 70.5%, training time by 28.1%, and peak GPU memory by 28.6%.
Limitations and Future Work
The authors noted that gains were more modest on image-audio and image-text benchmarks compared to video-audio. This likely reflects weaker task-relevant correspondence between those specific modality pairs.
Future work could involve:
- Incorporating category-level agreement to better handle cases where some classes are highly transferable while others are modality-locked.
- Developing more sophisticated decomposition constraints tailored to specific types of transfer (e.g., temporal vs. static).
Summary updated