Abstract
In the field of audio classification, Convolutional Neural Networks (CNNs) and Audio Spectrogram Transformers (ASTs) represent two dominant yet structurally distinct paradigms. This paper introduces Cross-Model Knowledge Distillation (CMKD), a training framework where these architectures serve as mutual teachers. By distilling knowledge across models with different inductive biases—spatial locality in CNNs and global context in Transformers—CMKD enables student models to significantly outperform their teachers and set new state-of-the-art benchmarks on FSD50K, AudioSet, and ESC-50.
Introduction
Audio classification has traditionally relied on CNNs due to their inherent inductive biases, such as spatial locality and translation equivariance, which are well-suited for spectrogram analysis. Recently, Transformers like the AST have emerged as powerful alternatives, capable of learning global relationships directly from data. However, Transformers often require massive datasets and are computationally intensive, while CNNs may struggle with long-range dependencies. CMKD addresses this by using Knowledge Distillation (KD) to transfer the unique strengths of one architecture to the other without modifying the underlying model structures.
The CMKD Framework
The proposed CMKD method utilizes bi-directional distillation, exploring both $CNN \rightarrow AST$ and $AST \rightarrow CNN$ pathways. Unlike standard KD, which typically involves a larger teacher and a smaller student of the same architecture, CMKD focuses on the cross-pollination of structural advantages.
Key technical requirements for successful CMKD include consistent data augmentation (ensuring teacher and student see identical views of the data) and a balanced loss function combining standard Binary Cross-Entropy with Kullback–Leibler divergence between model logits. This approach requires no architectural changes, making it highly versatile for existing pipelines.
Experimental Results
The authors evaluated CMKD across three major benchmarks: FSD50K, AudioSet, and ESC-50. The results consistently showed that CMKD-trained models outperformed their baseline counterparts trained only on ground-truth labels.
On the FSD50K dataset, CMKD achieved a state-of-the-art 62.9 mAP, a substantial improvement over the previous best of 57.6. Similar gains were observed on AudioSet (47.1 mAP) and ESC-50 (96.8% accuracy). A notable finding is the "Weak Teacher" paradox: even a smaller, less accurate teacher can improve a larger student if their inductive biases differ, suggesting that the student learns a new perspective rather than just mimicking the teacher's output.
Key Insights and Property Transfer
Probing tests revealed that CMKD facilitates a genuine transfer of properties. Transformers trained via CNN distillation became more robust to time-shifting (inheriting translation invariance), while CNNs trained via AST distillation showed improved resilience to noise and frequency masking (inheriting a more global perspective). Visualizations of attention maps further confirmed that KD from a CNN forces Transformers to capture global information at much earlier layers than standard training.
Conclusion
CMKD proves that CNNs and Transformers can be highly effective teachers for one another. By bridging the gap between local and global feature extraction through distillation, the framework achieves superior performance and efficiency. Future work may explore the application of CMKD in other domains, such as computer vision, where the synergy between these two architectures remains a promising area of research.
Summary updated