Abstract
Traditional speech enhancement (SE) models primarily rely on acoustic features, often failing to leverage the linguistic context that humans naturally use to interpret noisy speech. This paper presents Cross-Modality Knowledge Transfer (CMKT), a novel framework that integrates linguistic knowledge from pre-trained Large Language Models (LLMs) into SE models during the training phase. By employing a dual-branch architecture and a specialized misalignment strategy, CMKT "teaches" the SE model to recognize semantic structures without requiring text input or LLMs during inference. Experimental results on Mandarin and English corpora demonstrate significant improvements in speech quality (PESQ) and intelligibility (STOI) across various signal-to-noise ratios (SNRs).
Introduction
Speech enhancement aims to extract clean speech from noisy environments. While acoustic-only models have seen significant progress, they lack the "linguistic common sense" that allows humans to reconstruct missing information based on grammar and semantics. Existing "text-informed" SE models typically require transcripts during inference, which is impractical for real-time or real-world applications. Furthermore, aligning discrete text tokens with continuous acoustic signals remains a technical challenge.
The CMKT framework addresses these issues by using LLMs as linguistic teachers only during training. This approach ensures that the final deployed model remains lightweight and purely audio-based while benefiting from high-level semantic representations.
Methodology
The CMKT framework utilizes a dual-branch architecture to bridge the gap between acoustic and linguistic modalities.
Architecture
The system consists of two main components:
- SE Branch: A standard SE backbone (such as Conformer, Transformer, or BLSTM) that processes noisy log-amplitude spectrograms to produce enhanced outputs.
- CMKT Module: During training, a frozen, pre-trained LLM (e.g., BERT, LLaMA-2) extracts linguistic embeddings from ground-truth text. A Cross-Modality Transformer (CMT) is then used to align the SE model’s internal acoustic features with these linguistic embeddings.
Misalignment Strategy and Loss
A critical innovation in CMKT is the misalignment strategy. Rather than enforcing strict 1:1 temporal alignment, the authors introduce controlled temporal shifts (left and right). This forces the SE model to learn context-aware representations by looking beyond local patterns to find linguistic matches.
The model is optimized using a joint loss function:
- MSE Loss: Measures the reconstruction error between enhanced and clean spectrograms.
- CMA Loss (Cosine Similarity): Maximizes the similarity between acoustic embeddings and target linguistic embeddings from the LLM.
Experimental Setup
The framework was evaluated on two major datasets:
- Mandarin: AISHELL-1 (150 hours).
- English: LibriSpeech (360 hours). Noisy versions were created by mixing clean speech with various noise types at SNRs ranging from -15 dB to 15 dB.
Results and Discussion
The CMKT framework consistently outperformed baseline models across all tested architectures and metrics.
Key Findings
- Performance Gains: On the AISHELL-1 dataset, the CMKT-enhanced Conformer improved the PESQ score from 1.898 to 2.113. Similar gains were observed in STOI, particularly in low-SNR environments.
- LLM Selection: Interestingly, bidirectional models like BERT often outperformed generative models like LLaMA-2, likely because the non-causal nature of BERT aligns better with standard SE architectures.
- Robustness: The framework proved effective even when trained with imperfect, ASR-generated transcripts, indicating that "perfect" linguistic knowledge is not a prerequisite for improvement.
- Inference Efficiency: Since the CMKT module is discarded after training, the inference speed and memory footprint are identical to standard SE models.
Conclusion
Linguistic Knowledge Transfer Learning provides a robust method for enhancing speech by leveraging the semantic power of LLMs. By shifting the linguistic requirement to the training stage, CMKT maintains practical utility for real-world audio-only applications. Future work will explore optimal LLM layer selection and the application of this framework to related tasks like speech separation and de-reverberation.
Summary updated