Abstract
Active speaker detection (ASD) is a critical component in human-machine interaction, yet many existing end-to-end frameworks struggle with high computational complexity and large input sizes, hindering real-time application. This paper introduces a novel two-stream end-to-end framework that fuses visual features from a VGG-M network with raw Mel Frequency Cepstrum Coefficients (MFCC) extracted from audio. By employing Bidirectional Gated Recurrent Units (BiGRU) to model temporal dynamics both before and after fusion, the proposed model achieves a balance between accuracy and speed. Experimental results on the AVA-ActiveSpeaker dataset demonstrate that this strategy is robust to noise and achieves an inference time of 44.41 ms, making it suitable for real-time use while maintaining a state-of-the-art accuracy of 88.929%.
Introduction
Active Speaker Detection (ASD) involves identifying which individuals in a video are speaking at any given time. This task is fundamental for applications such as automated video editing, speech enhancement, and human-robot interaction. Despite recent advances in deep learning, two primary challenges remain: the high computational cost of current models and a lack of exploration into diverse feature fusion strategies. Most existing frameworks rely on Convolutional Neural Networks (ConvNets) for both audio and visual streams, which can be inefficient. This work addresses these issues by proposing a streamlined architecture that prioritizes both robustness and inference speed.
Proposed Methodology
The core of the proposed approach is a two-stream end-to-end framework designed for efficiency and temporal awareness.
Framework Architecture
The architecture consists of two specialized streams:
- Visual Stream: Utilizes a modified VGG-M architecture. To capture motion information effectively, the network employs $5 \times 7 \times 7$ filters across a sequence of five consecutive face frames.
- Audio Stream: Instead of a heavy ConvNet, the model leverages MFCCs, which represent sound based on human auditory perception. Three variants were explored: raw MFCCs (M1), MFCCs processed through fully connected layers (M2), and MFCCs processed through a VGG-M network (M3).
Temporal Modeling and Fusion
To handle the temporal dynamics of speech and facial movement, each stream is followed by two layers of Bidirectional Gated Recurrent Units (BiGRUs). The outputs from both streams are then concatenated and passed through a final BiGRU layer to model the joint audiovisual relationship. A softmax layer provides the final classification.
Dataset and Evaluation
The model was evaluated on the AVA-ActiveSpeaker dataset, a large-scale benchmark containing 3.65 million labeled frames. This dataset is particularly challenging due to real-world conditions such as low-resolution imagery, occlusions, and varied speaker orientations.
Results and Discussion
The experimental results highlight the trade-offs between model complexity, accuracy, and robustness.
Performance and Speed
The best-performing model (M3) achieved an accuracy of 88.929%, comparable to state-of-the-art methods. Crucially, the M1 model (using raw MFCCs) achieved an inference time of 44.41 ms, well within the requirements for real-time processing. Even the more complex M3 model maintained an inference time under 48 ms.
Robustness to Noise
A key finding of this research is the robustness of raw MFCC features. When subjected to Gaussian noise, the M1 model demonstrated superior stability compared to models that used additional ConvNet layers for audio processing. This suggests that for ASD, simpler audio representations can be more resilient in noisy environments.
Conclusion
This work presents a high-performance, real-time-capable framework for Active Speaker Detection. By combining VGG-M visual features with MFCC audio features and leveraging BiGRUs for temporal context, the model achieves state-of-the-art results with significantly reduced inference times. Future work will explore more sophisticated audio features and evaluate performance on mobile and edge hardware to further extend the practical utility of this framework.
Summary updated