Voice Activity Detection by Upper Body Motion Analysis and Unsupervised Domain Adaptation

Muhammad Shahid, Cigdem Beyan, Vittorio MurinoWorkshops @ ICCV 2019 · OpenReview

Abstract

We present a novel vision-based voice activity detection (VAD) method that relies only on automatic upper body motion (UBM) analysis. Traditionally, VAD is performed using audio features, but visual cues are desirable when audio is unavailable due to technical, ethical, or legal constraints. Psychology literature confirms that motion patterns during speech differ significantly from those during silence. However, individual and cultural variations in body language create a "domain shift" between training and test data. To address this, we combine stacked sparse autoencoders and subspace alignment to learn domain-invariant features. Our approach is person-invariant, requiring no re-training for new individuals, and achieves results superior to state-of-the-art video-only methods on real-life datasets.

Introduction

Determining "who is speaking and when"—Voice Activity Detection—is a fundamental task in multimedia analysis. While typically handled via audio, traditional VAD struggles in noisy "cocktail party" environments or privacy-sensitive scenarios. Existing visual VAD methods often focus on lip-reading, which requires high-resolution frontal views and fails under occlusion. This research proposes using upper body motion (gestures and posture shifts) as a robust alternative. The primary challenge lies in the diversity of human movement; a model trained on one person's gestures may not generalize to another. This study introduces a pipeline that leverages deep learning and unsupervised domain adaptation to bridge this gap.

Methodology

The proposed system follows a multi-stage pipeline designed to extract motion features and align them across different subjects without requiring labeled data from the target domain.

Overview of the proposed vision-based VAD pipeline including dynamic image construction and domain adaptation.

1. Dynamic Image Construction

To capture temporal information in a static format suitable for standard Convolutional Neural Networks (CNNs), the system uses "dynamic images." This technique compresses the motion of 10 consecutive video frames into a single RGB image. These images highlight the evolution of gestures and posture over time.

Comparison of dynamic images for speaking (top) and non-speaking (bottom) states, showing distinct motion signatures.

2. Feature Extraction and Domain Adaptation

A ResNet50 architecture, fine-tuned on these dynamic images, extracts high-level features. To handle the person-to-person variability in motion, the authors implement Unsupervised Domain Adaptation:

  • Stacked Sparse Autoencoder (2AE): Learns a robust, non-linear representation common to both the training (source) and test (target) data.
  • Subspace Alignment (SA): Uses PCA-based alignment to map the source feature space onto the target person's feature space, ensuring the classifier operates on a consistent distribution.

3. Classification

A linear Support Vector Machine (SVM) is trained on these adapted features to distinguish between "speaking" and "non-speaking" states.

Experimental Results

The method was evaluated on the Columbia VAD dataset, a real-life 87-minute panel discussion. The authors modified the dataset by expanding bounding boxes to include the full upper body, capturing more gestural information than head-only crops.

Key findings include:

  • High Accuracy: The system achieved an average F1-score of 89.20%, significantly outperforming previous video-only benchmarks (78%–85%).
  • Competitive with Multimodal: Remarkably, this motion-only approach performed on par with state-of-the-art multimodal systems (89.54% F1) that utilize both audio and lip-syncing.
  • Consistency: The low standard deviation in performance (5.14%) across different speakers demonstrates the effectiveness of the domain adaptation in making the model person-invariant.

Conclusion

This research demonstrates that upper body motion is a powerful, often overlooked signal for detecting speech. By utilizing dynamic images and unsupervised domain adaptation, the proposed method provides a robust, "deaf" VAD solution that generalizes across different individuals. Future work will focus on reducing temporal latency and adapting the system for more complex, crowded environments.

Summary updated

Abstract

We present a novel vision-based voice activity detection (VAD) method that relies only on automatic upper body motion (UBM) analysis. Traditionally, VAD is performed using audio features only, but the use of visual cues instead of audio can be desirable especially when audio is not available such as due to technical, ethical or legal issues. Psychology literature confirms that the way people move while speaking is different from while they are not speaking. This motivates us to claim that an effective representation of UBM can be used to detect "Who is Speaking and When". On the other hand, the way people move during their speech varies a lot from culture to culture, and even person to person in the same culture. This results in unrelated UBM representations, such that the distribution of training and test data becomes disparate. To overcome this, we combine stacked sparse autoencoders and simple subspace alignment methods while a classifier is jointly learned using the VAD labels of the training data only. This yields new domain invariant feature representations for training and test data, showing improved VAD results. Our approach is applicable to any person without requiring re-training. The tests applied on a publicly available real-life VAD dataset show better results as compared to the state-of-the-art video-only VAD methods. Moreover, the ablation study justifies the superiority of the proposed method and demonstrates the positive contribution of each component.
BibTeX
@inproceedings{shahid2019voice,
  author = {Muhammad Shahid and Cigdem Beyan and Vittorio Murino},
  title = {{Voice Activity Detection by Upper Body Motion Analysis and Unsupervised Domain Adaptation}},
  booktitle = {Workshops @ ICCV},
  year = {2019},
  url = {https://openreview.net/forum?id=wXV5RkpaAO},
}
Figure