Linguistic Knowledge Transfer Learning for Speech Enhancement

Kuo-Hsuan Hung, Xugang Lu, Szu-Wei Fu, Huan-Hsin Tseng, Hsin-Yi Lin, Chii-Wann Lin, Yu TsaoarXiv 2025 · DOI · arXiv

Abstract

Traditional speech enhancement (SE) models primarily rely on acoustic features, often failing to leverage the linguistic context that humans naturally use to interpret noisy speech. This paper presents Cross-Modality Knowledge Transfer (CMKT), a novel framework that integrates linguistic knowledge from pre-trained Large Language Models (LLMs) into SE models during the training phase. By employing a dual-branch architecture and a specialized misalignment strategy, CMKT "teaches" the SE model to recognize semantic structures without requiring text input or LLMs during inference. Experimental results on Mandarin and English corpora demonstrate significant improvements in speech quality (PESQ) and intelligibility (STOI) across various signal-to-noise ratios (SNRs).

Introduction

Speech enhancement aims to extract clean speech from noisy environments. While acoustic-only models have seen significant progress, they lack the "linguistic common sense" that allows humans to reconstruct missing information based on grammar and semantics. Existing "text-informed" SE models typically require transcripts during inference, which is impractical for real-time or real-world applications. Furthermore, aligning discrete text tokens with continuous acoustic signals remains a technical challenge.

The CMKT framework addresses these issues by using LLMs as linguistic teachers only during training. This approach ensures that the final deployed model remains lightweight and purely audio-based while benefiting from high-level semantic representations.

Methodology

The CMKT framework utilizes a dual-branch architecture to bridge the gap between acoustic and linguistic modalities.

Architecture

The system consists of two main components:

  1. SE Branch: A standard SE backbone (such as Conformer, Transformer, or BLSTM) that processes noisy log-amplitude spectrograms to produce enhanced outputs.
  2. CMKT Module: During training, a frozen, pre-trained LLM (e.g., BERT, LLaMA-2) extracts linguistic embeddings from ground-truth text. A Cross-Modality Transformer (CMT) is then used to align the SE model’s internal acoustic features with these linguistic embeddings.
The CMKT framework architecture featuring the SE branch and the CMKT learning module.

Misalignment Strategy and Loss

A critical innovation in CMKT is the misalignment strategy. Rather than enforcing strict 1:1 temporal alignment, the authors introduce controlled temporal shifts (left and right). This forces the SE model to learn context-aware representations by looking beyond local patterns to find linguistic matches.

The model is optimized using a joint loss function:

  • MSE Loss: Measures the reconstruction error between enhanced and clean spectrograms.
  • CMA Loss (Cosine Similarity): Maximizes the similarity between acoustic embeddings and target linguistic embeddings from the LLM.

Experimental Setup

The framework was evaluated on two major datasets:

  • Mandarin: AISHELL-1 (150 hours).
  • English: LibriSpeech (360 hours). Noisy versions were created by mixing clean speech with various noise types at SNRs ranging from -15 dB to 15 dB.

Results and Discussion

The CMKT framework consistently outperformed baseline models across all tested architectures and metrics.

Performance comparison of various SE backbones with and without CMKT across different SNR conditions.

Key Findings

  • Performance Gains: On the AISHELL-1 dataset, the CMKT-enhanced Conformer improved the PESQ score from 1.898 to 2.113. Similar gains were observed in STOI, particularly in low-SNR environments.
  • LLM Selection: Interestingly, bidirectional models like BERT often outperformed generative models like LLaMA-2, likely because the non-causal nature of BERT aligns better with standard SE architectures.
  • Robustness: The framework proved effective even when trained with imperfect, ASR-generated transcripts, indicating that "perfect" linguistic knowledge is not a prerequisite for improvement.
  • Inference Efficiency: Since the CMKT module is discarded after training, the inference speed and memory footprint are identical to standard SE models.

Conclusion

Linguistic Knowledge Transfer Learning provides a robust method for enhancing speech by leveraging the semantic power of LLMs. By shifting the linguistic requirement to the training stage, CMKT maintains practical utility for real-world audio-only applications. Future work will explore optimal LLM layer selection and the application of this framework to related tasks like speech separation and de-reverberation.

Summary updated

Abstract

Linguistic knowledge plays a crucial role in spoken language comprehension. It provides essential semantic and syntactic context for speech perception in noisy environments. However, most speech enhancement (SE) methods predominantly rely on acoustic features to learn the mapping relationship between noisy and clean speech, with limited exploration of linguistic integration. While text-informed SE approaches have been investigated, they often require explicit speech-text alignment or externally provided textual data, constraining their practicality in real-world scenarios. Additionally, using text as input poses challenges in aligning linguistic and acoustic representations due to their inherent differences. In this study, we propose the Cross-Modality Knowledge Transfer (CMKT) learning framework, which leverages pre-trained large language models (LLMs) to infuse linguistic knowledge into SE models without requiring text input or LLMs during inference. Furthermore, we introduce a misalignment strategy to improve knowledge transfer. This strategy applies controlled temporal shifts, encouraging the model to learn more robust representations. Experimental evaluations demonstrate that CMKT consistently outperforms baseline models across various SE architectures and LLM embeddings, highlighting its adaptability to different configurations. Additionally, results on Mandarin and English datasets confirm its effectiveness across diverse linguistic conditions, further validating its robustness. Moreover, CMKT remains effective even in scenarios without textual data, underscoring its practicality for real-world applications. By bridging the gap between linguistic and acoustic modalities, CMKT offers a scalable and innovative solution for integrating linguistic knowledge into SE models, leading to substantial improvements in both intelligibility and enhancement performance.
BibTeX
@misc{hung2025linguistic,
  author = {Kuo-Hsuan Hung and Xugang Lu and Szu-Wei Fu and Huan-Hsin Tseng and Hsin-Yi Lin and Chii-Wann Lin and Yu Tsao},
  title = {{Linguistic Knowledge Transfer Learning for Speech Enhancement}},
  year = {2025},
  eprint = {2503.07078},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2503.07078},
}
Figure