Abstract
Active speaker detection (ASD) in multimodal environments is crucial for various applications, from video conferencing to human-robot interaction. This paper introduces FabuLight-ASD, an advanced ASD model that integrates facial, audio, and body pose information to enhance detection accuracy and robustness. Our model builds upon the existing Light-ASD framework by incorporating human pose data, represented through skeleton graphs, which minimises computational overhead. Using the Wilder Active Speaker Detection (WASD) dataset, renowned for reliable face and body bounding box annotations, we demonstrate FabuLight-ASD's effectiveness in real-world scenarios. Achieving an overall mean average precision (mAP) of 94.3%, FabuLight-ASD outperforms Light-ASD, which has an overall mAP of 93.7% across various challenging scenarios. The incorporation of body pose information shows a particularly advantageous impact, with notable improvements in mAP observed in scenarios with speech impairment, face occlusion, and human voice background noise. Furthermore, efficiency analysis indicates only a modest increase in parameter count (27.3%) and multiply-accumulate operations (up to 2.4%), underscoring the model's efficiency and feasibility. These findings validate the efficacy of FabuLight-ASD in enhancing ASD performance through the integration of body pose data.
Introduction
Active Speaker Detection (ASD)—the task of identifying who is speaking in a video—is a foundational component for human-robot interaction and intelligent video conferencing systems. While traditional systems rely heavily on facial movements (lip-syncing) and audio signals, they often struggle in "in-the-wild" conditions. When a speaker is distant from the camera, their face is occluded, or significant background noise is present, standard facial and audio cues become unreliable.
While some research has explored "contextual" models that analyze the entire scene, these are often computationally expensive and prone to overfitting. FabuLight-ASD addresses these challenges by leveraging body language (gestures and upper-body movements) as a vital, underutilized cue for speech activity that can be captured efficiently.
Methods: The FabuLight-ASD Architecture
FabuLight-ASD builds upon the Light-ASD framework, optimized for efficiency on embedded systems like social robots. The primary innovation is the introduction of a Body Pose Stream alongside traditional face and audio processing.
The model employs a three-stream architecture:
- Face Stream: Processes 3D stacks of grayscale face crops.
- Audio Stream: Analyzes 2D maps of Mel-frequency cepstrum coefficients (MFCCs).
- Body Pose Stream: Utilizes a sequence of 2D coordinates representing body joints (skeletons).
To maintain efficiency, the body stream uses Spatial-Temporal Graph Convolutional Networks (ST-GCN). Rather than processing raw pixels, the model treats the human skeleton as a graph, capturing how joints move relative to each other spatially and how those movements evolve over time. Features from all three encoders are fused via element-wise summation and passed through a Bidirectional Gated Recurrent Unit (BiGRU) for the final frame-level prediction. Auxiliary training heads for face and body modalities ensure robust feature learning without adding inference-time overhead.
Experimental Setup and Datasets
The model was evaluated using the Wilder Active Speaker Detection (WASD) dataset. WASD was selected over the common AVA-ActiveSpeaker dataset due to its superior bounding box accuracy and reliable body position annotations. WASD includes 30 hours of video across five challenging real-world scenarios:
- Optimal Conditions (OC): Clear visibility and minimal overlap.
- Speech Impairment (SI): Frequent speech overlap.
- Face Occlusion (FO): Scenarios where the mouth or face is hidden.
- Human Voice Noise (HVN): Significant background chatter.
- Surveillance Settings (SS): Low resolution and distant subjects.
Key Insights and Results
The integration of body pose data proved most beneficial when primary facial cues were compromised. FabuLight-ASD achieved an overall 94.3% mAP, surpassing the baseline Light-ASD (93.7%).
Robustness in Challenging Scenarios
Significant gains were observed in the "Face Occlusion" and "Human Voice Noise" categories. In HVN scenarios, the model's performance improved from 84.7% to 86.4% mAP. This suggests that body pose helps the model "anchor" audio signals to the correct individual, using physical gestures to disambiguate whose voice is being heard amidst background noise.
Efficiency and Design
Despite the added modality, FabuLight-ASD remains remarkably efficient:
- Model Size: Only 1.3 million parameters.
- Computational Overhead: The body stream adds only 2.4% to the total multiply-accumulate operations (MACs), totaling 209 million MACs per frame—far lower than billions required by "contextual" models.
- Joint Selection: Experiments revealed that using only the upper body (11 joints) often performed as well as or better than the whole body (17 joints), indicating that lower-body movement can be "noise" for speech detection.
Conclusion and Future Work
FabuLight-ASD demonstrates that body language is a powerful and computationally cheap cue for enhancing active speaker detection, especially in adverse conditions. While it shows marked improvements, performance in surveillance settings (SS) remains an area for growth, currently trailing behind heavier models like TalkNet.
Future research will focus on adaptive weighting, allowing the model to dynamically prioritize the body stream when facial or audio signals are detected as low-quality. The team also plans to deploy FabuLight-ASD on physical social robots to evaluate real-time performance in complex, multi-party human interactions.
Code and model weights are available at https://github.com/knowledgetechnologyuhh/FabuLight-ASD.
Summary updated