TALD: Transferability-Aware Logit Disentanglement for Cross-Modal Knowledge Distillation

Hui Li, Pengfei Yang, Xinyi Feng, Junhong Liu, Tianyang Zheng, Hui Zeng, Le Dong, Kaibing Zhang, Mengting Zhao, Quan WangarXiv 2026 · arXiv

TL;DR

TALD (Transferability-Aware Logit Disentanglement) is a new framework that prevents "negative transfer" in cross-modal knowledge distillation by separating modality-general knowledge from modality-specific noise. By working directly in the class-aligned logit space, it achieves superior accuracy gains across vision, audio, and text benchmarks while remaining significantly more efficient than previous feature-based methods.

What Problem Does This Paper Address?

In Cross-Modal Knowledge Distillation (CMKD), we try to teach a student model in one modality (e.g., audio) using a pre-trained teacher from another (e.g., video). The hope is that the teacher's rich representations will help the student learn faster or better.

However, modalities aren't identical. A video teacher might rely on visual cues (like lip movement) that simply don't exist for an audio student. If the teacher forces these "modality-specific" cues onto the student, it can lead to negative transfer—where the student actually performs worse than if it had just trained on its own.

Figure 1: Class-structured negative transfer and the transferability- aware view (RAVDESS, V→A; single run). The visual and audio baselines differ in per-class accuracy by up to about 42 percentage points (pp), and the KD student falls below the audio baseline on several classes (e.g., C2, C4). Conventional KD uses the full teacher response, while TALD constructs its distillation target by adaptively fusing learned modality-general components from the teacher and reference student.

The challenge is: how do we extract only the "modality-general" knowledge (the stuff that applies to both) while ignoring the modality-specific "noise"?

Methods

The authors propose TALD, a two-stage framework that performs disentanglement in the logit space (the final class scores before softmax). Logits are "class-aligned," meaning the $i$-th dimension always represents the $i$-th class, regardless of modality, making them a compact and direct space for modeling transferability.

Stage 1: Transferability-Aware Disentanglement

TALD uses a Residual Disentangler to split logits into two parts:

  1. Modality-General (GEN): Information useful across modalities.
  2. Modality-Specific (SPEC): Information unique to the source modality.

This split is guided by two clever criteria:

  • C1 (Self-benefiting): The SPEC component should help classification within its own modality.
  • C2 (Cross-modal exclusivity): Adding the SPEC component from one modality to the GEN logits of another should increase error (cross-entropy), proving it's truly modality-specific.
Figure 2: Overview of the TALD framework. In the first stage, the Disentangler decomposes teacher and reference student logits into modality-general and modality-specific components, while the Modulator predicts sample-specific margins and contribution-ratio bounds to regulate the decomposition. In the second stage, the modality-general components are fused using weights derived from the predicted margins and distilled to the target-modality student.

To handle the fact that some samples are easier to transfer than others, a Sample-Adaptive Modulator predicts a per-sample "margin." This margin acts as a control signal, relaxing or tightening the disentanglement constraints based on the specific input.

Stage 2: Adaptive Knowledge Distillation

Once the GEN components are learned, TALD creates a "clean" distillation target by adaptively fusing the GEN logits from the teacher and a "reference student" (a model trained only on the target modality). This ensures the final student receives supervision that is both expert-informed and modality-appropriate.

Figure 3: Architecture of the Residual Disentangler. Left: An MLP shared across the two modalities predicts the SPEC components, and the GEN components are obtained by residual subtraction. Right: A direction-specific CMP applies a class-conditioned affine transformation to a SPEC component to construct the cross-modal contribution used in the Stage-1 C2 constraint.

Datasets

The authors evaluated TALD across five diverse multimodal benchmarks:

  • AV-MNIST: Image and audio of handwritten digits.
  • RAVDESS: Video and audio for emotion recognition.
  • VGGSound-50k: Video and audio for scene classification.
  • CrisisMMD-V2: Image and text for humanitarian crisis classification.
  • Fakeddit: Image and text for fine-grained fake news detection.

Key Insights and Ideas

  • Logits are enough: Most prior work tries to filter high-dimensional features, which is computationally expensive. TALD proves that disentangling at the logit level is not only more efficient but also more effective because it directly targets the classification task.
  • The "Residual" Trick: By defining the modality-general component as the original logit minus a learned specific component ($z_{gen} = z - z_{spec}$), the model maintains an exact additive relationship, ensuring the disentanglement is grounded in the original predictions.
  • Exclusivity as a Filter: The idea that "specific" info should actively hurt a different modality (C2 criterion) is a powerful inductive bias for ensuring the GEN component is truly universal.

Key Results

  • Superior Accuracy: TALD achieved the largest mean accuracy gain (+1.38 percentage points) over the "no distillation" baseline across all ten transfer directions.
  • Reliability: It showed a 84% positive-transfer frequency, meaning it almost always helped the student, whereas baseline methods often suffered from negative transfer (improving in only 10% to 70% of cases).
  • Efficiency: Compared to the state-of-the-art MGDFR, TALD reduced total FLOPs by 70.5%, training time by 28.1%, and peak GPU memory by 28.6%.

Limitations and Future Work

The authors noted that gains were more modest on image-audio and image-text benchmarks compared to video-audio. This likely reflects weaker task-relevant correspondence between those specific modality pairs.

Future work could involve:

  • Incorporating category-level agreement to better handle cases where some classes are highly transferable while others are modality-locked.
  • Developing more sophisticated decomposition constraints tailored to specific types of transfer (e.g., temporal vs. static).

Summary updated

Abstract

Cross-modal knowledge distillation can incur negative transfer when source-modality supervision conflicts with the target modality's discriminative characteristics. We propose TALD, a two-stage framework for Transferability-Aware Logit Disentanglement in class-aligned logit space. Guided by task-dependent self-benefiting and cross-modal exclusivity criteria, a Residual Disentangler decomposes logits from a teacher and an independently trained target-modality reference student into modality-general and modality-specific components. A Sample-Adaptive Modulator regulates decomposition constraints and sample-specific fusion weights. The learned modality-general components are then adaptively fused to supervise the final student. Across five benchmarks spanning vision, audio, and text, TALD achieves the largest mean accuracy gain and highest positive-transfer frequency among the compared methods. Across ten transfer directions with five runs each, TALD improves over training without distillation in 84% of evaluations, compared with at most 70% for the baselines, while offering a favorable accuracy-efficiency trade-off. We plan to release the code.
BibTeX
@misc{li2026tald,
  author = {Hui Li and Pengfei Yang and Xinyi Feng and Junhong Liu and Tianyang Zheng and Hui Zeng and Le Dong and Kaibing Zhang and Mengting Zhao and Quan Wang},
  title = {{TALD: Transferability-Aware Logit Disentanglement for Cross-Modal Knowledge Distillation}},
  year = {2026},
  eprint = {2611.25126},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.25126},
}
Figure