Abstract
Detecting speech in crowded social environments is a complex task, often hindered by acoustic noise or privacy restrictions. This paper explores "No-audio Multimodal Speech Detection" by leveraging physical movement signals. We propose a dual-stream deep learning architecture that fuses triaxial accelerometer data with overhead video. Our model, utilizing 1D-CNNs, C3D, and Bi-directional Gated Recurrent Units (Bi-GRUs), achieves a state-of-the-art Area Under the Curve (AUC) of 0.712 on the MediaEval 2020 dataset, outperforming existing unimodal and multimodal benchmarks.
Introduction
In social signal processing, identifying active speakers is fundamental to understanding group dynamics. While audio-based Voice Activity Detection (VAD) is standard, it is susceptible to environmental noise and privacy concerns. This study addresses the "No-audio" challenge, focusing on non-verbal cues—such as head movements, gestures, and postural shifts—that accompany speech. The primary difficulty lies in effectively synchronizing and fusing disparate data streams: 2D visual motion from above and 3D acceleration from a neck-worn sensor.
Methodology
The proposed system employs a dual-stream architecture designed to extract temporal features from each modality before integration.
AccelNet (Accelerometer Stream)
The accelerometer stream processes raw triaxial data through three 1D-convolutional layers. These layers are configured with specific strides to establish a receptive field of approximately one second. A Bi-directional Gated Recurrent Unit (Bi-GRU) follows, capturing the temporal dependencies of physical movements across time.
VideoNet (Visual Stream)
The visual stream utilizes a pre-trained C3D (Convolutional 3D) network to extract motion patterns from overhead video. Video is processed in one-second segments, with features subsequently passed through a Bi-GRU to maintain temporal context.
Multimodal Fusion
As illustrated in the architecture diagram, features from both AccelNet and VideoNet are concatenated into a single vector. This "early fusion" representation is processed through fully connected (FC) layers and a sigmoid activation function to determine the final probability of speech.
Experimental Results
The model was evaluated using the MediaEval 2020 No-audio Multimodal Speech Detection dataset, comprising data from 70 subjects. Performance was measured using the Area Under the Curve (AUC).
The proposed fusion model achieved an AUC of 0.712, establishing a new state-of-the-art for this task. Notably, the unimodal accelerometer performance (0.689) significantly exceeded that of the video stream (0.656), suggesting that wearable sensors capture speech-related vibrations and neck movements more effectively than overhead cameras.
Discussion and Conclusion
The integration of Bi-GRUs proved essential for modeling speech as a sequential process rather than an instantaneous event. By aligning both modalities to a one-second feature vector, the fusion process successfully leveraged the complementary strengths of each sensor.
Despite the overall success, performance variability across subjects remains a challenge. Future work will focus on dynamic weighting of modalities to handle cases where one signal may be noisier than the other, as well as fine-tuning the visual backbone specifically for social interaction cues.
Summary updated