Abstract
This paper introduces DLAV-SE, a diffusion-based speech enhancement (SE) system that integrates audio, visual (lip movements), and linguistic (semantic) modalities. By utilizing a Cross-Modal Knowledge Transfer (CMKT) mechanism during training, the model learns to align audio-visual representations with semantic context from a pretrained BERT model. This approach allows the system to benefit from linguistic awareness during inference without requiring text inputs or additional computational overhead. Evaluations on Mandarin and English datasets, as well as real-world driving scenarios, demonstrate that DLAV-SE significantly outperforms existing audio-visual baselines in speech quality and intelligibility.
Introduction
Traditional speech enhancement (SE) methods primarily rely on audio signals, which often fail in adverse acoustic environments characterized by heavy background noise or overlapping speakers. Audio-Visual SE (AVSE) improves robustness by incorporating lip-reading cues. However, human communication also relies heavily on linguistic context—the meaning of what is being said.
Integrating discrete linguistic data with continuous audio-visual signals presents a challenge due to significant dimensional and distributional gaps. Furthermore, requiring real-time text transcription during inference would introduce unacceptable latency. DLAV-SE addresses these issues by using linguistic information as a training-time constraint to guide the model's latent space, achieving "free" semantic awareness at runtime.
Proposed Architecture
The DLAV-SE framework is a hybrid predictive-generative diffusion model operating on complex spectrograms. It consists of two main stages: a predictive stage that provides an initial estimate and a generative stage that refines the output through a diffusion process.
Multi-Modal Integration
- Visual Modality: Lip region embeddings are extracted via a pretrained ResNet-18 and fused into the diffusion process using cross-attention.
- Linguistic Modality: During training, transcripts are processed by a pretrained BERT model to extract context-dependent semantic features. If transcripts are unavailable, a Whisper ASR model is used to generate them.
- Cross-Modal Knowledge Transfer (CMKT): To bridge the gap between modalities, the authors introduce an alignment module using either Multi-Head Cross-Attention (MHCA) or Optimal Transport (OT). This forces the audio-visual latent space to match the linguistic latent space of BERT.
Experimental Results
The model was evaluated on the TMSV (Mandarin), AVSEC-3 (English), and a new real-world dataset, AS-CarSpeech.
Quantitative Performance
On the TMSV dataset, the full Audio+Visual+Linguistic (A+V+L) model achieved a PESQ of 1.74 and SI-SDR of 4.1, significantly outperforming the audio-only baseline (PESQ 1.54). On the AVSEC-3 benchmark, the OT-variant reached a PESQ of 1.94, surpassing the challenge baseline of 1.49 and other state-of-the-art models like MMDTN.
Qualitative Analysis and Insights
Spectrogram analysis reveals that the linguistic modality specifically aids in reconstructing high-frequency speech components and suppressing residual noise during silent intervals. The semantic data essentially teaches the model the expected "shape" of valid speech.
Conclusion
DLAV-SE demonstrates that linguistic information can be effectively distilled into audio-visual SE models during training to improve performance without increasing inference cost. Future work will focus on reducing training complexity and testing the framework's generalization across larger, more diverse acoustic benchmarks.
Summary updated