Abstract: Improving the performance of on-device audio classification models remains a challenge given the computational limits of the mobile environment. Many studies leverage knowledge distillation to boost predictive performance by transferring the knowledge from large models to on-device models. However, most lack a mechanism to distill the essence of the temporal information, which is crucial to audio classification tasks, or similar architecture is often required. In this paper, we propose a new knowledge distillation method designed to incorporate the temporal knowledge embedded in attention weights of large transformer-based models into on-device models. Our distillation method is applicable to various types of architectures, including the non-attention-based architectures such as CNNs or RNNs, while retaining the original network architecture during inference. Through extensive experiments on both an audio event detection dataset and a noisy keyword spotting dataset, we show that our proposed method improves the predictive performance across diverse on-device architectures.
Introduction
Running complex audio classification tasks, such as keyword spotting or audio event detection, directly on mobile devices requires highly efficient, lightweight models. To bridge the performance gap between these small models and massive state-of-the-art models, researchers often use Knowledge Distillation (KD)—training a small "student" model to mimic a large "teacher" model.
However, standard KD usually only transfers final output probabilities (logits). In audio tasks, temporal information—knowing exactly when a sound occurred in a clip—is crucial, but this is typically lost in logit distillation. While some methods distill attention maps to preserve this data, they often require the student model to also be a transformer, which is computationally heavy for edge devices. This paper proposes a framework to transfer temporal knowledge from a large transformer to standard, non-transformer architectures (like CNNs and RNNs) without adding any computational overhead during inference.
Proposed Method: Temporal Knowledge Distillation
The proposed framework employs a novel approach to align the temporal focus of the student with that of the teacher:
- Teacher Model: A massive, pre-trained transformer (XLSR-wav2vec 2.0) is used as the teacher. Using "attention rollout," a unified attention map $a_T$ is extracted to represent the model's temporal focus.
- Student Model: Various lightweight on-device models are tested, including LSTM, CNN, and CRNN architectures.
- Attention Distillation: For student models without native attention, a temporary attention layer is attached during the training phase.
- Alignment: The training objective minimizes the Kullback-Leibler (KL) divergence between the teacher's attention weights and the student's temporary attention weights. Linear interpolation matches dimensions if sequence lengths differ.
Crucially, once training is complete, the temporary attention layer is discarded for non-attention models. The student retains the learned temporal focus while running at its original, optimized speed during inference.
Experimental Results
The method was evaluated on two primary datasets: FSD50K (a multi-label audio event detection dataset) and a newly constructed Noisy Speech Commands v2 dataset, designed to test temporal localization by embedding keywords into longer noisy background clips (2s to 8s).
Performance Gains
The proposed method improved the Mean Average Precision (mAP) across all tested on-device architectures on FSD50K. Relative improvements were substantial: LSTM-P (+13.9%), TC-ResNet (+7.6%), CRNN (+9.5%), Att-RNN (+21.5%), and MHAtt-RNN (+25.3%).
Temporal Alignment
A key finding is that distilled student models successfully learn to focus on relevant temporal segments. As shown in Figure 3, both attention-based and non-attention-based students align their focus with the teacher's attention peak at the keyword location.
On the Noisy Speech Commands v2 dataset, distilled models consistently outperformed vanilla models across all noise lengths. Remarkably, some attention-based student models (Att-RNN, MHAtt-RNN) even outperformed the massive wav2vec 2.0 teacher, achieving over 91% accuracy on 8-second clips where the teacher scored 90.95%.
Conclusion
Temporal awareness can be decoupled from specific architectures. By incorporating a transformer-based teacher's temporal focus during training, simpler models can achieve significant performance boosts. This "zero-overhead" distillation is highly effective for on-device applications, particularly as audio clip length and background noise increase.
Summary updated