Abstract
We present a novel vision-based voice activity detection (VAD) method that relies only on automatic upper body motion (UBM) analysis. Traditionally, VAD is performed using audio features, but visual cues are desirable when audio is unavailable due to technical, ethical, or legal constraints. Psychology literature confirms that motion patterns during speech differ significantly from those during silence. However, individual and cultural variations in body language create a "domain shift" between training and test data. To address this, we combine stacked sparse autoencoders and subspace alignment to learn domain-invariant features. Our approach is person-invariant, requiring no re-training for new individuals, and achieves results superior to state-of-the-art video-only methods on real-life datasets.
Introduction
Determining "who is speaking and when"—Voice Activity Detection—is a fundamental task in multimedia analysis. While typically handled via audio, traditional VAD struggles in noisy "cocktail party" environments or privacy-sensitive scenarios. Existing visual VAD methods often focus on lip-reading, which requires high-resolution frontal views and fails under occlusion. This research proposes using upper body motion (gestures and posture shifts) as a robust alternative. The primary challenge lies in the diversity of human movement; a model trained on one person's gestures may not generalize to another. This study introduces a pipeline that leverages deep learning and unsupervised domain adaptation to bridge this gap.
Methodology
The proposed system follows a multi-stage pipeline designed to extract motion features and align them across different subjects without requiring labeled data from the target domain.
1. Dynamic Image Construction
To capture temporal information in a static format suitable for standard Convolutional Neural Networks (CNNs), the system uses "dynamic images." This technique compresses the motion of 10 consecutive video frames into a single RGB image. These images highlight the evolution of gestures and posture over time.
2. Feature Extraction and Domain Adaptation
A ResNet50 architecture, fine-tuned on these dynamic images, extracts high-level features. To handle the person-to-person variability in motion, the authors implement Unsupervised Domain Adaptation:
- Stacked Sparse Autoencoder (2AE): Learns a robust, non-linear representation common to both the training (source) and test (target) data.
- Subspace Alignment (SA): Uses PCA-based alignment to map the source feature space onto the target person's feature space, ensuring the classifier operates on a consistent distribution.
3. Classification
A linear Support Vector Machine (SVM) is trained on these adapted features to distinguish between "speaking" and "non-speaking" states.
Experimental Results
The method was evaluated on the Columbia VAD dataset, a real-life 87-minute panel discussion. The authors modified the dataset by expanding bounding boxes to include the full upper body, capturing more gestural information than head-only crops.
Key findings include:
- High Accuracy: The system achieved an average F1-score of 89.20%, significantly outperforming previous video-only benchmarks (78%–85%).
- Competitive with Multimodal: Remarkably, this motion-only approach performed on par with state-of-the-art multimodal systems (89.54% F1) that utilize both audio and lip-syncing.
- Consistency: The low standard deviation in performance (5.14%) across different speakers demonstrates the effectiveness of the domain adaptation in making the model person-invariant.
Conclusion
This research demonstrates that upper body motion is a powerful, often overlooked signal for detecting speech. By utilizing dynamic images and unsupervised domain adaptation, the proposed method provides a robust, "deaf" VAD solution that generalizes across different individuals. Future work will focus on reducing temporal latency and adapting the system for more complex, crowded environments.
Summary updated