Bridging The Multi-Modality Gaps of Audio, Visual and Linguistic for Speech Enhancement

Meng-Ping Lin, Jen-Cheng Hou, Chia-Wei Chen, Shao-Yi Chien, Jun-Cheng Chen, Xugang Lu, Yu TsaoarXiv 2025 · DOI · arXiv

Abstract

This paper introduces DLAV-SE, a diffusion-based speech enhancement (SE) system that integrates audio, visual (lip movements), and linguistic (semantic) modalities. By utilizing a Cross-Modal Knowledge Transfer (CMKT) mechanism during training, the model learns to align audio-visual representations with semantic context from a pretrained BERT model. This approach allows the system to benefit from linguistic awareness during inference without requiring text inputs or additional computational overhead. Evaluations on Mandarin and English datasets, as well as real-world driving scenarios, demonstrate that DLAV-SE significantly outperforms existing audio-visual baselines in speech quality and intelligibility.

Introduction

Traditional speech enhancement (SE) methods primarily rely on audio signals, which often fail in adverse acoustic environments characterized by heavy background noise or overlapping speakers. Audio-Visual SE (AVSE) improves robustness by incorporating lip-reading cues. However, human communication also relies heavily on linguistic context—the meaning of what is being said.

Integrating discrete linguistic data with continuous audio-visual signals presents a challenge due to significant dimensional and distributional gaps. Furthermore, requiring real-time text transcription during inference would introduce unacceptable latency. DLAV-SE addresses these issues by using linguistic information as a training-time constraint to guide the model's latent space, achieving "free" semantic awareness at runtime.

Proposed Architecture

The DLAV-SE framework is a hybrid predictive-generative diffusion model operating on complex spectrograms. It consists of two main stages: a predictive stage that provides an initial estimate and a generative stage that refines the output through a diffusion process.

The DLAV-SE training architecture featuring the Predictive and Generative UNets, BERT-based linguistic integration, and the Cross-Modal Knowledge Transfer (CMKT) module.

Multi-Modal Integration

  • Visual Modality: Lip region embeddings are extracted via a pretrained ResNet-18 and fused into the diffusion process using cross-attention.
  • Linguistic Modality: During training, transcripts are processed by a pretrained BERT model to extract context-dependent semantic features. If transcripts are unavailable, a Whisper ASR model is used to generate them.
  • Cross-Modal Knowledge Transfer (CMKT): To bridge the gap between modalities, the authors introduce an alignment module using either Multi-Head Cross-Attention (MHCA) or Optimal Transport (OT). This forces the audio-visual latent space to match the linguistic latent space of BERT.

Experimental Results

The model was evaluated on the TMSV (Mandarin), AVSEC-3 (English), and a new real-world dataset, AS-CarSpeech.

Quantitative Performance

On the TMSV dataset, the full Audio+Visual+Linguistic (A+V+L) model achieved a PESQ of 1.74 and SI-SDR of 4.1, significantly outperforming the audio-only baseline (PESQ 1.54). On the AVSEC-3 benchmark, the OT-variant reached a PESQ of 1.94, surpassing the challenge baseline of 1.49 and other state-of-the-art models like MMDTN.

Qualitative Analysis and Insights

Spectrogram analysis reveals that the linguistic modality specifically aids in reconstructing high-frequency speech components and suppressing residual noise during silent intervals. The semantic data essentially teaches the model the expected "shape" of valid speech.

Spectrogram comparisons across different modality combinations. The A+V+L model (f) shows the clearest reconstruction of high-frequency components compared to the noisy (a) and audio-only (c) versions.

Conclusion

DLAV-SE demonstrates that linguistic information can be effectively distilled into audio-visual SE models during training to improve performance without increasing inference cost. Future work will focus on reducing training complexity and testing the framework's generalization across larger, more diverse acoustic benchmarks.

Summary updated

Abstract

Speech enhancement (SE) aims to improve the quality and intelligibility of speech in noisy environments. Recent studies have shown that incorporating visual cues in audio signal processing can enhance SE performance. Given that human speech communication naturally involves audio, visual, and linguistic modalities, it is reasonable to expect additional improvements by integrating linguistic information. However, effectively bridging these modality gaps, particularly during knowledge transfer remains a significant challenge. In this paper, we propose a novel multi-modal learning framework, termed DLAV-SE, which leverages a diffusion-based model integrating audio, visual, and linguistic information for audio-visual speech enhancement (AVSE). Within this framework, the linguistic modality is modeled using a pretrained language model (PLM), which transfers linguistic knowledge to the audio-visual domain through a cross-modal knowledge transfer (CMKT) mechanism during training. After training, the PLM is no longer required at inference, as its knowledge is embedded into the AVSE model through the CMKT process. We conduct a series of SE experiments to evaluate the effectiveness of our approach. Results show that the proposed DLAV-SE system significantly improves speech quality and reduces generative artifacts, such as phonetic confusion, compared to state-of-the-art (SOTA) methods. Furthermore, visualization analyses confirm that the CMKT method enhances the generation quality of the AVSE outputs. These findings highlight both the promise of diffusion-based methods for advancing AVSE and the value of incorporating linguistic information to further improve system performance.
BibTeX
@misc{lin2025bridging,
  author = {Meng-Ping Lin and Jen-Cheng Hou and Chia-Wei Chen and Shao-Yi Chien and Jun-Cheng Chen and Xugang Lu and Yu Tsao},
  title = {{Bridging The Multi-Modality Gaps of Audio, Visual and Linguistic for Speech Enhancement}},
  year = {2025},
  eprint = {2501.13375},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2501.13375},
}
Figure