Abstract
In this paper, we propose a visual embedding approach to improving embedding aware speech enhancement (EASE) by synchronizing visual lip frames at the phone and place of articulation levels. We first extract visual embedding from lip frames using a pre-trained phone or articulation place recognizer for visual-only EASE (VEASE). Next, we extract audio-visual embedding from noisy speech and lip videos in an information intersection manner, utilizing a complementarity of audio and visual features for multi-modal EASE (MEASE). Experiments on the TCD-TIMIT corpus corrupted by simulated additive noises show that our proposed subword based VEASE approach is more effective than conventional embedding at the word level. Moreover, visual embedding at the articulation place level, leveraging upon a high correlation between place of articulation and lip shapes, shows an even better performance than that at the phone level. Finally the proposed MEASE framework, incorporating both audio and visual embedding, yields significantly better speech quality and intelligibility than those obtained with the best visual-only and audio-only EASE systems.
Introduction
Speech enhancement is a critical component for technologies like hearing aids, smart assistants, and automatic speech recognition (ASR). While deep learning has significantly improved audio-only enhancement, these systems often struggle in "cocktail party" scenarios—extremely noisy environments with multiple speakers. Humans naturally solve this by lip-reading, a phenomenon known as the McGurk effect.
Existing Audio-Visual Speech Enhancement (AVSE) systems try to mimic this by using video of a speaker’s mouth. However, many current models use word-level embeddings, which are often too coarse to capture the rapid, subtle lip movements that correspond to specific sounds. Furthermore, raw video contains redundant information, such as lighting changes, that does not contribute to speech reconstruction. This research proposes a more precise classification target for visual data that correlates better with the physical mechanics of speech: the subword level.
Methodology
The researchers introduce the Embedding Aware Speech Enhancement (EASE) framework, which operates in two primary modes: Visual-only (VEASE) and Multi-modal (MEASE).
Visual Embedding Extractor (VEASE)
The core of the VEASE approach is the extraction of visual embeddings from lip frames using a pre-trained recognizer. Unlike traditional systems that target entire words, this model shifts to subword targets—specifically phones (distinct linguistic sounds) and the place of articulation (the physical position of the tongue or lips, such as "labial" for 'p' or 'b').
As shown in Figure 1, the extractor utilizes a 3D-Convolutional layer followed by a ResNet-18 backbone to process temporal lip sequences. The resulting features are passed through a Bidirectional Gated Recurrent Unit (BiGRU) and a SoftMax layer to produce posteriors for words, phones, or places of articulation. The intermediate "Visual Embedding" is then used to guide the speech enhancement process.
Multi-modal EASE (MEASE)
The MEASE model extends this by fusing audio (FBANK features) and visual embeddings. The authors advocate for early fusion, where audio and visual features are integrated during the embedding extraction phase rather than later in the enhancement network. This "information intersection" approach uses a shared BiGRU to learn a joint representation supervised by articulation place labels, preserving more useful information for final speech reconstruction.
Experimental Setup
The models were evaluated using the TCD-TIMIT corpus, a high-quality audio-visual dataset featuring 59 speakers with phonetic-level transcriptions. For comparison, the LRW (Lip Reading in the Wild) dataset was used to train word-level baselines. To test robustness, the researchers used NOISEX-92 to provide unseen noise types, such as Destroyer Engine and Speech Babble, simulating unpredictable real-world environments.
Key Insights and Results
The "Place of Articulation" Advantage
A primary finding of this study is that lip shapes correlate much more strongly with the place of articulation than with specific phones. For instance, the lip shapes for 'p', 'b', and 'm' (labial) are nearly identical. By training the model to recognize these broader categories, the visual embedding becomes more robust and less susceptible to classification errors.
Granularity and Performance
Word-level embeddings were found to be slow to converge and often missed the fine-grained timing necessary to mask noise effectively. Moving to the subword level allows the model to track the speech signal frame-by-frame. Objective metrics, including PESQ (Perceptual Evaluation of Speech Quality) and STOI (Short-Time Objective Intelligibility), confirmed this:
- Subword Superiority: The
VEASE-placemodel consistently outperformed theVEASE-wordbaseline. - MEASE Performance: The multi-modal MEASE model achieved the best results. At a noisy -5 dB SNR, the audio-only baseline (NoEASE) scored 2.07 on PESQ, while MEASE reached 2.29.
- Stability: Unlike many AVSE models that degrade speech in clean environments, MEASE remained stable at 15 dB SNR, achieving a PESQ of 3.42 compared to 3.21 for audio-only systems.
Conclusion
This work demonstrates that synchronizing lip movements with subword-level linguistic units, particularly the place of articulation, significantly enhances the effectiveness of audio-visual speech enhancement. While the current supervised approach relies on painstakingly labeled datasets like TCD-TIMIT, future work will explore self-supervised techniques to learn these correlations from raw, unlabeled video, potentially enabling training on much larger "in-the-wild" datasets.
Summary updated