S-VVAD: Visual Voice Activity Detection by Motion Segmentation

Muhammad Shahid, Cigdem Beyan, Vittorio MurinoWACV 2021 · OpenReview

Abstract

We address the challenging Voice Activity Detection (VAD) problem, which determines "Who is Speaking and When?" in audiovisual recordings. The typical audio-based VAD systems can be ineffective in the presence of ambient noise or noise variations. Moreover, due to technical or privacy reasons, audio might not be always available. In such cases, the use of video modality to perform VAD is desirable. Almost all existing visual VAD methods rely on body part detection, e.g., face, lips, or hands. In contrast, we propose a novel visual VAD method operating directly on the entire video frame, without the explicit need of detecting a person or his/her body parts. Our method, named S-VVAD, learns body motion cues associated with speech activity within a weakly supervised segmentation framework. Therefore, it not only detects the speakers/not-speakers but simultaneously localizes the image positions of them. It is an end-to-end pipeline, person-independent and it does not require any prior knowledge nor pre-processing. S-VVAD performs well in various challenging conditions and demonstrates the state-of-the-art results on multiple datasets. Moreover, the better generalization capability of S-VVAD is confirmed for cross-dataset and person-independent scenarios.

Introduction

Voice Activity Detection (VAD) is a critical component in applications ranging from smart assistants to automated meeting transcription. Traditionally, VAD systems rely on audio signals, which can be unreliable in noisy environments or when audio is unavailable due to technical or privacy constraints. Visual VAD (VVAD) offers a robust alternative. However, most existing VVAD methods depend on detecting specific body parts, such as lips or faces. This reliance makes them susceptible to failure when subjects are occluded, wearing masks, or turned away from the camera.

S-VVAD (Visual Voice Activity Detection by Motion Segmentation) addresses these limitations by proposing a holistic approach. Instead of focusing on isolated body parts, S-VVAD analyzes the entire video frame to identify motion cues associated with speech—such as head tilts, shoulder movements, and gestures. This method is person-independent, requires no pre-processing like face tracking, and operates within a weakly supervised segmentation framework.

Methodology

The S-VVAD pipeline consists of two primary phases: training and inference.

Training via Weak Supervision

The training process is divided into two stages, C1 and C2, to leverage image-level labels for pixel-wise segmentation.

  1. Motion Representation: The system converts 10 consecutive RGB frames into a "Dynamic Image" (DI). This single image captures the temporal changes and motion patterns within the sequence, reducing computational load while preserving essential spatio-temporal cues.
  2. Stage C1 (Classification): A ResNet50 architecture is fine-tuned using "sub-dynamic images" to classify frames as "speaking" or "not-speaking."
  3. Stage C2 (Mask Generation & FCN Training): Class Activation Maps (CAMs) are extracted from the fine-tuned ResNet50 to identify the regions contributing most to the classification. These CAMs serve as weak labels to train a Fully Convolutional Network (FCN) for pixel-wise segmentation.
The S-VVAD training pipeline: Stage C1 fine-tunes a ResNet50 for classification, while Stage C2 uses Class Activation Maps (CAMs) to generate masks for training a segmentation FCN.

Inference and Localization

During the inference phase, the trained FCN processes a full-frame Dynamic Image to generate a segmentation map. This map identifies regions as background, speaking, or not-speaking. To provide final speaker localization, the system applies Affinity Propagation clustering to these regions, allowing it to project bounding boxes onto the original RGB frames.

The S-VVAD inference pipeline: The trained FCN produces a pixel-wise prediction from a Dynamic Image, which is then clustered to localize speakers with bounding boxes.

Experimental Evaluation

The performance of S-VVAD was evaluated across three datasets featuring real-world panel discussions:

  • Columbia Dataset: A standard benchmark with five panelists.
  • Modified Columbia Dataset: A more complex version where multiple panelists appear in the same frame, testing the system's ability to distinguish between concurrent speakers.
  • RealVAD Dataset: An "in-the-wild" dataset featuring nine panelists, significant occlusions, side views, and non-speech movements like drinking water.

Results and Discussion

S-VVAD demonstrates state-of-the-art performance and significant robustness:

  • Accuracy: On the Columbia dataset, the system achieved a 93.97% F1-score, outperforming existing methods with higher reliability across different individuals.
  • Multi-Speaker Handling: In the Modified Columbia dataset, S-VVAD significantly improved upon previous benchmarks, reaching an 87.30% F1-score compared to the previous best of 80.40%.
  • Generalization: Cross-dataset testing confirmed that S-VVAD learns general motion patterns of speech rather than dataset-specific features, maintaining competitive performance even when trained and tested on different environments.

A key insight from this work is that speech is a "whole-body" activity. By moving beyond lip-reading, S-VVAD remains effective even when the face is partially obscured. Furthermore, the use of weak supervision proves that pixel-perfect manual masks are unnecessary for effective speaker localization.

Conclusion

S-VVAD provides a robust, end-to-end solution for visual voice activity detection. By treating speaker detection as a semantic segmentation task based on holistic body motion, it overcomes the brittleness of traditional body-part-dependent methods. While the system shows strong generalization, future work will focus on closing the domain gap in highly unconstrained environments and exploring multi-camera configurations to further enhance accuracy.

Summary updated

Abstract

We address the challenging Voice Activity Detection (VAD) problem, which determines "Who is Speaking and When?" in audiovisual recordings. The typical audio-based VAD systems can be ineffective in the presence of ambient noise or noise variations. Moreover, due to technical or privacy reasons, audio might not be always available. In such cases, the use of video modality to perform VAD is desirable. Almost all existing visual VAD methods rely on body part detection, e.g., face, lips, or hands. In contrast, we propose a novel visual VAD method operating directly on the entire video frame, without the explicit need of detecting a person or his/her body parts. Our method, named S-VVAD, learns body motion cues associated with speech activity within a weakly supervised segmentation framework. Therefore, it not only detects the speakers/not-speakers but simultaneously localizes the image positions of them. It is an end-to-end pipeline, person-independent and it does not require any prior knowledge nor pre-processing. S-VVAD performs well in various challenging conditions and demonstrates the state-of-the-art results on multiple datasets. Moreover, the better generalization capability of S-VVAD is confirmed for cross-dataset and person-independent scenarios.
BibTeX
@inproceedings{shahid2021s,
  author = {Muhammad Shahid and Cigdem Beyan and Vittorio Murino},
  title = {{S-VVAD: Visual Voice Activity Detection by Motion Segmentation}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2021},
  url = {https://openreview.net/forum?id=GquaoZ8-tpX},
}
Figure