Robust Audio-Visual Speech Enhancement: Correcting Misassignments in Complex Environments With Advanced Post-Processing

Wenze Ren, Kuo-Hsuan Hung, Rong Chao, You-Jin Li, Hsin-Min Wang, Yu Tsao

Abstract

This paper addresses the prevalent issue of incorrect speech output in audio-visual speech enhancement (AVSE) systems, which is often caused by poor video quality and mismatched training and test data. We introduce a post-processing classifier (PPC) to rectify these erroneous outputs, ensuring that the enhanced speech corresponds accurately to the intended speaker. We also adopt a mixup strategy in PPC training to improve its robustness. Experimental results on the AVSE-challenge dataset show that integrating PPC into the AVSE model can significantly improve AVSE performance, and combining PPC with the AVSE model trained with permutation invariant training (PIT) yields the best performance. The proposed method substantially outperforms the baseline model by a large margin. This work highlights the potential for broader applications across various modalities and architectures, providing a promising direction for future research in this field.

Introduction

Speech enhancement (SE) aims to extract a target speaker's voice from noisy environments. While traditional audio-only methods are effective, Audio-Visual Speech Enhancement (AVSE) leverages visual cues, such as lip movements, to improve performance in challenging acoustic scenarios. However, AVSE systems often suffer from "misassignment"—a phenomenon where the model incorrectly enhances an interfering speaker instead of the target, particularly when video quality is degraded or noise conditions are mismatched. This paper introduces a Post-Processing Classifier (PPC) to act as a multimodal verification step, ensuring the enhanced audio aligns with the visual input.

Methodology

The proposed system follows a two-stage approach: a backbone enhancement network followed by a post-processing verification module.

Backbone: TF-Gridnet

The primary separation engine is TF-Gridnet, a sophisticated architecture that processes audio across three dimensions: intra-frame spectral patterns, sub-band temporal changes, and full-band global context via self-attention. This allows for robust feature extraction from noisy speech signals.

Overall architecture of the proposed AVSE system, featuring TF-Gridnet and the Post-Processing module.

Post-Processing Classifier (PPC)

The PPC is designed to validate whether the output of the enhancement stage matches the target speaker's visual features. Three variants were explored to determine the most effective way to fuse audio and visual embeddings.

Architectural variants of the Post-Processing Classifier (PPC). PPC-3 utilizes cosine similarity for optimal verification.

PPC-3 emerged as the most effective variant. By explicitly calculating the frame-wise cosine similarity between modalities, it can accurately detect misassignments. If the similarity is low, the system assumes the interference was enhanced and can swap the outputs accordingly. To improve robustness, the authors employed mixup training, creating synthetic examples to help the PPC generalize to unseen noise.

Training Strategy: PIT Synergy

An important innovation is the use of Permutation Invariant Training (PIT) for the AVSE backbone. PIT allows the model to output either the target or the interference without penalty during training. This results in "cleaner" errors—where the model produces a clear version of the wrong speaker—which the PPC can then easily identify and correct. This synergy significantly outperformed models forced to output only the target.

Experimental Results

The system was evaluated on the LRS3 dataset and the AVSE Challenge (2023 & 2024) benchmarks. The proposed method demonstrated substantial improvements over standard baselines:

  • Signal Quality: On the Eva2023 evaluation set, the proposed system achieved an SI-SDR of 7.94 dB, a significant gain over the baseline performance of 3.67 dB.
  • Classification Accuracy: The PPC-3 model reached 99.5% accuracy on development data and maintained over 94% accuracy on challenging test sets.
  • Challenge Performance: In the 2024 AVSE challenge, the addition of the PPC module provided a 1.91 dB SI-SDR improvement over the standalone enhancement model.

Conclusion

This work demonstrates that a dedicated post-processing "sanity check" can significantly mitigate the misassignment problem in AVSE. By combining the strengths of TF-Gridnet with a similarity-based classifier and PIT-based training, the system achieves state-of-the-art performance in complex environments. Future research will explore extending this verification framework to other modalities, such as bone-conduction sensors, to further enhance robustness in extreme conditions.

Summary updated

BibTeX
@inproceedings{ren2024robust,
  author = {Wenze Ren and Kuo-Hsuan Hung and Rong Chao and You-Jin Li and Hsin-Min Wang and Yu Tsao},
  title = {{Robust Audio-Visual Speech Enhancement: Correcting Misassignments in Complex Environments With Advanced Post-Processing}},
  booktitle = {O-cocosda},
  year = {2024},
  doi = {10.1109/o-cocosda64382.2024.10800033},
}
Figure