Audio-Visual Speech Enhancement in Noisy Environments via Emotion-Based Contextual Cues

Tassadaq Hussain, Kia Dashtipour, Yu Tsao, Amir HussainarXiv 2024 · OpenReview

Abstract

In real-world environments, background noise significantly degrades the intelligibility and clarity of human speech. Audio-visual speech enhancement (AVSE) attempts to restore speech quality, but existing methods often fall short, particularly in dynamic noise conditions. This study investigates the inclusion of emotion as a novel contextual cue within AVSE, hypothesizing that incorporating emotional understanding can improve speech enhancement performance. We propose a novel emotion-aware AVSE system that leverages both auditory and visual information. It extracts emotional features from the facial landmarks of the speaker and fuses them with corresponding audio and visual modalities. This enriched data serves as input to a deep UNet-based encoder-decoder network, specifically designed to orchestrate the fusion of multimodal information enhanced with emotion. The network iteratively refines the enhanced speech representation through an encoder-decoder architecture, guided by perceptually-inspired loss functions for joint learning and optimization. We train and evaluate the model on the CMU Multimodal Opinion Sentiment and Emotion Intensity (CMU-MOSEI) dataset, a rich repository of audio-visual recordings with annotated emotions. Our comprehensive evaluation demonstrates the effectiveness of emotion as a contextual cue for AVSE. By integrating emotional features, the proposed system achieves significant improvements in both objective and subjective assessments of speech quality and intelligibility, especially in challenging noise environments. Compared to baseline AVSE and audio-only speech enhancement systems, our approach exhibits a noticeable increase in PESQ and STOI, indicating higher perceptual quality and intelligibility. Large-scale listening tests corroborate these findings, suggesting improved human understanding of enhanced speech.

Introduction

Speech Enhancement (SE) is the process of isolating human voice from background noise. While deep learning has advanced this field, traditional audio-only methods often struggle in non-stationary environments, such as crowded cafes or windy streets, where noise is unpredictable. Audio-Visual (AVSE) methods have improved performance by incorporating visual cues like lip-reading. However, even these systems can falter in extreme noise.

This research introduces emotion as a contextual cue, arguing that a speaker's emotional state fundamentally alters their vocal intensities and facial tensions. By teaching a model to recognize how a person is feeling, the system can more accurately predict what their "clean" speech should sound like, effectively reducing the search space for denoising algorithms.

Methodology

The proposed Emotion-Aware AVSE system utilizes a deep UNet-based encoder-decoder architecture. The pipeline integrates three distinct processing branches:

  1. Audio Branch: Processes the magnitude spectrum of noisy speech using Short-Time Fourier Transform (STFT) followed by convolutional layers.
  2. Visual Branch: Employs MTCNN for face detection, followed by a ResNet-18 and a Temporal Convolutional Network (TCN) to capture the spatiotemporal dynamics of the speaker's face.
  3. Emotion Branch: This "secret sauce" uses a pre-trained EfficientNet model to extract high-dimensional emotional embeddings from facial landmarks, identifying nuances across eight primary emotions (e.g., Happiness, Sadness, Anger).

These three modalities are concatenated into a composite tensor and fed into a UNet decoder that generates an "Estimated Magnitude Mask," which is applied to the original noisy audio to produce the enhanced version.

The proposed emotion-aware AVSE architecture, featuring integrated audio, visual, and emotion processing branches.

Experimental Results

The model was trained and evaluated on the CMU-MOSEI dataset, which contains over 23,500 video segments. To simulate real-world stress, clean audio was corrupted with noise from benchmarks like DNS and DEMAND across a wide range of Signal-to-Noise Ratios (SNR).

As shown in the comparative analysis, the proposed $UNet_{E-AVSE}$ system significantly outperforms audio-only baselines and state-of-the-art AVSE models like CochleaNet and IO-AVSE across all objective metrics:

  • PESQ (Quality): Improved from 1.326 (noisy baseline) to 1.685.
  • STOI (Intelligibility): Increased from 0.447 to 0.703.
  • SDI (Distortion): Achieved a marked reduction in speech distortion (1.994) compared to the noisy input (3.061).
Objective evaluation metrics comparing the proposed UNet_E-AVSE against various baseline and state-of-the-art frameworks.

Further optimization using a Modulation-loss function yielded the best balance for human perception, achieving a STOI of 0.710. This suggests that perceptually-inspired training is superior to simple magnitude-wise loss for speech intelligibility.

Conclusion

Integrating emotional features provides a powerful contextual bridge that makes multimodal fusion more robust. While adding emotion increases model complexity, it significantly boosts the intelligibility of speech in high-noise scenarios. Future work will focus on fine-tuning the emotion extraction branch and exploring continuous emotional scales (Valence and Arousal) to provide even more granular context for denoising.

Summary updated

Abstract

In real-world environments, background noise significantly degrades the intelligibility and clarity of human speech. Audio-visual speech enhancement (AVSE) attempts to restore speech quality, but existing methods often fall short, particularly in dynamic noise conditions. This study investigates the inclusion of emotion as a novel contextual cue within AVSE, hypothesizing that incorporating emotional understanding can improve speech enhancement performance. We propose a novel emotion-aware AVSE system that leverages both auditory and visual information. It extracts emotional features from the facial landmarks of the speaker and fuses them with corresponding audio and visual modalities. This enriched data serves as input to a deep UNet-based encoder-decoder network, specifically designed to orchestrate the fusion of multimodal information enhanced with emotion. The network iteratively refines the enhanced speech representation through an encoder-decoder architecture, guided by perceptually-inspired loss functions for joint learning and optimization. We train and evaluate the model on the CMU Multimodal Opinion Sentiment and Emotion Intensity (CMU-MOSEI) dataset, a rich repository of audio-visual recordings with annotated emotions. Our comprehensive evaluation demonstrates the effectiveness of emotion as a contextual cue for AVSE. By integrating emotional features, the proposed system achieves significant improvements in both objective and subjective assessments of speech quality and intelligibility, especially in challenging noise environments. Compared to baseline AVSE and audio-only speech enhancement systems, our approach exhibits a noticeable increase in PESQ and STOI, indicating higher perceptual quality and intelligibility. Large-scale listening tests corroborate these findings, suggesting improved human understanding of enhanced speech.
BibTeX
@misc{hussain2024audio,
  author = {Tassadaq Hussain and Kia Dashtipour and Yu Tsao and Amir Hussain},
  title = {{Audio-Visual Speech Enhancement in Noisy Environments via Emotion-Based Contextual Cues}},
  year = {2024},
  eprint = {2402.16394},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2402.16394},
}
Figure