Correlating subword articulation with lip shapes for embedding aware audio-visual speech enhancement

Hang Chen, Jun Du, Yu Hu, Li-Rong Dai, Bao-Cai Yin, Chin-Hui Lee

TL;DR

This paper introduces a novel audio-visual speech enhancement framework that synchronizes lip movements with subword-level linguistic units, specifically the "place of articulation." By fusing these fine-grained visual cues with noisy audio at an early stage, the researchers achieved significant gains in speech quality and intelligibility, outperforming traditional word-level and audio-only systems.

What Problem Does This Paper Address?

Speech enhancement is a critical component for technologies like hearing aids, smart assistants, and automatic speech recognition (ASR). While deep learning has significantly improved audio-only enhancement, these systems often struggle in "cocktail party" scenarios—extremely noisy environments with multiple speakers.

Humans naturally solve this by lip-reading (the McGurk effect). Existing Audio-Visual Speech Enhancement (AVSE) systems try to mimic this by using video of a speaker’s mouth. However, many current models use word-level embeddings, which are often too coarse to capture the rapid, subtle lip movements that correspond to specific sounds. Furthermore, raw video contains a lot of "redundant" information (like lighting changes) that doesn't help with speech. The authors set out to find a more precise "classification target" for visual data that correlates better with the actual physics of speech.

Methods

The authors propose the Embedding Aware Speech Enhancement (EASE) framework, which comes in two flavors:

  1. VEASE (Visual-only EASE): This model extracts visual embeddings from lip frames using a pre-trained recognizer. The "aha" moment here is the shift from word-level targets to subword targets—specifically phones (the distinct sounds in a language) and the place of articulation (where the tongue or lips are positioned, e.g., "labial" for 'p' or 'b').
  2. MEASE (Multi-modal EASE): This model takes it a step further by fusing audio (FBANK features) and visual embeddings. They use an "information intersection" approach, where a shared BiGRU (Bidirectional Gated Recurrent Unit) learns a joint representation supervised by the articulation place labels.

Crucially, the authors argue for early fusion. Instead of concatenating audio and video features in the middle of the enhancement network (as many models do), they fuse them during the embedding extraction phase, which they found preserves more useful information for the final speech reconstruction.

Datasets

  • TCD-TIMIT: A high-quality audio-visual corpus featuring 59 speakers. It is unique because it includes phonetic-level transcriptions, making it ideal for training subword-level extractors.
  • LRW (Lip Reading in the Wild): Used to train the "word-level" baseline model for comparison.
  • NOISEX-92: Used to provide "unseen" noise types (like Destroyer Engine and Speech Babble) to test how well the model generalizes to real-world, unpredictable environments.

Key Insights and Ideas

  • The "Place of Articulation" Advantage: The researchers found that lip shapes correlate much more strongly with the place of articulation than with specific phones. For example, the lip shape for 'p', 'b', and 'm' (labial) is nearly identical. By training the model to recognize these broader categories, the visual embedding becomes more robust and less prone to "noise" in the classification task.
  • Granularity is the Key: Word-level embeddings are slow to converge and often miss the fine-grained timing needed to mask noise effectively. Moving to the subword level allows the model to "track" the speech signal frame-by-frame.
  • Complementarity: Audio and visual cues excel in different areas. Visual cues are "noise-immune" and help most when the signal-to-noise ratio (SNR) is very low. Audio embeddings help more when the SNR is higher. The MEASE model successfully balances these, showing improvements across all noise levels.

Key Results

The models were evaluated using PESQ (Perceptual Evaluation of Speech Quality) and STOI (Short-Time Objective Intelligibility).

  • Subword Superiority: The VEASE-place model consistently outperformed the VEASE-word baseline. Even without audio embeddings, simply choosing a better visual target improved results.
  • MEASE vs. The Rest: The multi-modal MEASE model was the clear winner. At a very noisy -5 dB SNR, the audio-only baseline (NoEASE) scored 2.07 on PESQ, while MEASE reached 2.29.
  • Stability at High SNR: A common failure of AVSE models is that they can actually degrade speech quality when the environment is already relatively clean (high SNR). The MEASE model avoided this, showing stable improvements even at 15 dB SNR, where it achieved a PESQ of 3.42 (compared to 3.21 for audio-only).

Limitations and Future Work

The primary limitation of this work is its reliance on supervised learning for the embedding extractors. Training the model requires datasets like TCD-TIMIT that have been painstakingly labeled at the phonetic level.

The authors suggest that future research should focus on unsupervised or self-supervised techniques. If a model could learn these effective visual-acoustic correlations from raw, unlabeled video, it could be trained on much larger "in-the-wild" datasets, potentially leading to even more generalized speech enhancement.

Summary updated

Abstract

In this paper, we propose a visual embedding approach to improve embedding aware speech enhancement (EASE) by synchronizing visual lip frames at the phone and place of articulation levels. We first extract visual embedding from lip frames using a pre-trained phone or articulation place recognizer for visual-only EASE (VEASE). Next, we extract audio-visual embedding from noisy speech and lip frames in an information intersection manner, utilizing a complementarity of audio and visual features for multi-modal EASE (MEASE). Experiments on the TCD-TIMIT corpus corrupted by simulated additive noises show that our proposed subword based VEASE approach is more effective than conventional embedding at the word level. Moreover, visual embedding at the articulation place level, leveraging upon a high correlation between place of articulation and lip shapes, demonstrates an even better performance than that at the phone level. Finally the experiments establish that the proposed MEASE framework, incorporating both audio and visual embeddings, yields significantly better speech quality and intelligibility than those obtained with the best visual-only and audio-only EASE systems.
BibTeX
@article{chen2021correlating,
  author = {Hang Chen and Jun Du and Yu Hu and Li-Rong Dai and Bao-Cai Yin and Chin-Hui Lee},
  title = {{Correlating subword articulation with lip shapes for embedding aware audio-visual speech enhancement}},
  journal = {Neural Networks},
  year = {2021},
  doi = {10.1016/j.neunet.2021.06.003},
}

Loading related content...

Figure