Abstract
In this paper, we propose an intra-inter representation fusion knowledge distillation (KD) framework with time-frequency calibration (I$^2$RF-TFCKD) for SE, which achieves distillation through the fusion of multi-layer teacher-student feature flows. Different from previous distillation strategies for SE, the proposed framework fully utilizes the time-frequency differential information of speech while promoting global knowledge flow. Firstly, we construct a collaborative distillation paradigm for intra-set and inter-set correlations. Within a correlated set, multi-layer teacher-student features are pairwise matched for calibrated distillation. Subsequently, we generate representative features from each correlated set through residual fusion to form the fused feature set that enables inter-set knowledge interaction. Secondly, we propose a multi-layer interactive distillation based on dual-stream time-frequency cross-calibration, which calculates the teacher-student similarity calibration weights in the time and frequency domains respectively and performs cross-weighting, thus enabling refined allocation of distillation contributions across different layers according to speech characteristics. The proposed distillation strategy is applied to the dual-path dilated convolutional recurrent network (DPDCRN) that ranked first in the SE track of the L3DAS23 challenge. To evaluate the effectiveness of I$^2$RF-TFCKD, we conduct experiments on both single-channel and multi-channel SE datasets. Objective evaluations demonstrate that the proposed KD strategy consistently and effectively improves the performance of the low-complexity student model and outperforms other distillation schemes.
1. Introduction
Deep learning has revolutionized Speech Enhancement (SE), but top-tier models often exceed 5 million parameters and 10 GFLOPs, making them too computationally expensive for real-time use on edge devices like hearing aids or mobile phones. Knowledge Distillation (KD) is a common solution to this problem, where a small "student" network learns from a large "teacher" network. However, existing KD methods for SE often fail to account for the unique dual nature of speech—its evolution over time and its structure across frequencies—and typically ignore global knowledge flow within the architecture.
2. Proposed Method
The authors propose the I$^2$RF-TFCKD framework, which introduces two primary innovations to bridge the performance gap between large and small models.
2.1 Intra-Inter Representation Fusion
The framework groups the network into functional "correlated sets" (Encoder, Decoder, and F-T processing modules).
- Intra-set Distillation: Within a correlated set, a single student layer can learn from multiple teacher layers simultaneously. This "many-to-one" or cross-level mapping allows for a more flexible transfer of hierarchical information compared to traditional 1:1 layer matching.
- Inter-set Distillation: The framework uses a Recursive Fusion mechanism to generate "representative features" for each set. These fused features are then distilled across different sets, ensuring that knowledge flows globally throughout the entire architecture.
2.2 Time-Frequency Calibration (TFCKD)
Recognizing that speech information is distributed differently in time and frequency, TFCKD calculates distillation weights for both domains independently. The Time Flow captures frame-level speech activity, while the Frequency Flow focuses on spectral details. By cross-weighting these, the framework dynamically identifies which teacher-student feature pairs are most semantically relevant, refining the distillation process to prioritize clean speech features over noise.
2.3 Backbone Architecture
The backbone used for this study is the DPDCRN (Dual-Path Dilated Convolutional Recurrent Network), which previously won the SE track of the L3DAS23 challenge. It consists of dilated convolutional blocks and specialized Frequency-Time (F-T) modules.
3. Results and Discussion
The framework was evaluated on the DNS Challenge (single-channel) and L3DAS23 Challenge (multi-channel) datasets. Key findings include:
- Extreme Compression: The student model contains only 0.6M parameters (roughly 17% of the 3.5M teacher) and requires significantly fewer FLOPs.
- High Fidelity: On the DNS dataset, the student achieved a PESQ score of 3.03, significantly higher than the undistilled student (2.81) and approaching the teacher's 3.16.
- Multi-channel Success: In the L3DAS23 benchmark, the method improved the student's PESQ by 0.218 and notably reduced the Word Error Rate (WER), proving its effectiveness for downstream tasks like speech recognition.
- Recursive Fusion as a Redundancy Filter: By recursively fusing features, the model preserves critical information while discarding redundant data, making the "lesson plan" easier for the small student to digest.
4. Conclusion
The I$^2$RF-TFCKD framework effectively compresses massive speech enhancement models into lightweight versions suitable for edge devices. By leveraging time-frequency duality and global feature fusion, it enables small models to achieve performance remarkably close to state-of-the-art teachers. Future research will explore task generalization to acoustic echo cancellation and multi-teacher distillation scenarios.
Summary updated