Abstract
Speech enhancement (SE) is increasingly deployed on resource-constrained edge devices, necessitating lightweight yet high-performing models. This paper introduces Distil-DCCRN, a compressed version of the Deep Complex Convolution Recurrent Network (DCCRN). By utilizing a novel Attention Transfer + Kullback-Leibler (AT-KL) distillation method, Distil-DCCRN learns effectively from a complex Uformer teacher. The proposed method addresses dimension mismatches in time and channel axes, allowing the student to outperform the original DCCRN with only 1.1M parameters—a 70% reduction.
Introduction
As speech enhancement moves toward "edge" applications like smartphones and hearing aids, the demand for computationally efficient models has grown. While architectures like the Deep Complex Convolution Recurrent Network (DCCRN) offer high-quality audio by processing complex-valued features, their parameter count often exceeds the limits of small-footprint devices.
Knowledge Distillation (KD) is a common strategy to compress these models by training a smaller "student" to mimic a larger "teacher." However, standard KD methods often fail when the teacher and student have different internal architectures or audio processing settings (e.g., different STFT hop lengths), leading to misaligned feature maps. Distil-DCCRN addresses these challenges through a specialized distillation framework.
Methodology: The AT-KL Approach
The primary innovation of this work is the AT-KL (Attention Transfer + Kullback-Leibler Divergence) distillation method. This approach allows a student model to learn from a teacher even when their internal data dimensions are mismatched.
The distillation process involves three key components:
- Attention Transfer (AT): To resolve dimension mismatches, the teacher’s high-dimensional features are compressed into 2D maps using an $l_2$-norm. This preserves the spatial "attention" of the features while making the shapes compatible between the teacher and student.
- KL Divergence: Instead of forcing the student to match exact numerical values, the compressed features are converted into probability distributions. The student then minimizes the KL divergence to mimic the teacher’s information distribution.
- Multi-Dimensional Alignment: This alignment is performed across both the time dimension (to handle varying audio frame rates) and the channel dimension (to handle different model widths).
Experimental Setup
The models were evaluated using the DNS (Deep Noise Suppression) Challenge dataset, a standard industry benchmark. The authors focused on the "no reverb" test set to compare Distil-DCCRN against various state-of-the-art (SOTA) models, including the original DCCRN and the transformer-based Uformer.
Key Results
Distil-DCCRN achieves significant efficiency gains without sacrificing audio quality. It uses only 1.1M parameters, representing a 70% reduction from the original DCCRN (3.74M). Furthermore, computational complexity (FLOPs) was reduced from 15.7G to 7.4G.
As shown in the results table, Distil-DCCRN outperformed the original DCCRN across multiple metrics:
- PESQ (Voice Quality): 2.80 vs. 2.74
- SI-SNR (Signal Integrity): 17.80 dB vs. 17.71 dB
- NB-PESQ: 3.31 vs. 3.26
These results demonstrate that the AT-KL distillation method allows a smaller model to not only mimic but potentially exceed the performance of its larger predecessor by learning more efficient feature representations.
Discussion and Insights
A critical finding of this research is that aligning the time dimension is as vital as aligning the channel dimension in audio distillation. Because different models often use different "hop lengths," temporal alignment ensures the student learns the correct temporal dynamics of the speech signal.
Furthermore, the success of distilling a Transformer-based teacher (Uformer) into a Convolutional/Recurrent student (DCCRN) proves the versatility of the AT-KL approach. While a performance gap remains between Distil-DCCRN and the massive Uformer teacher (3.02 PESQ), the student provides a highly competitive alternative for real-time, low-resource environments.
Conclusion
Distil-DCCRN represents a significant step forward in small-footprint speech enhancement. By focusing on information distribution rather than exact value replication, the AT-KL method enables effective cross-architecture knowledge transfer. Future work may explore applying these techniques to even smaller "tiny-ML" models for ultra-low-latency applications.
Summary updated