Abstract
Interactions involving children span a wide range of important domains from learning to clinical diagnostic and therapeutic contexts. Automated analyses of such interactions are motivated by the need to seek accurate insights and offer scale and robustness across diverse and wide-ranging conditions. Identifying the speech segments belonging to the child is a critical step in such modeling. Conventional child-adult speaker classification typically relies on audio modeling approaches, overlooking visual signals that convey speech articulation information, such as lip motion. Building on the foundation of an audio-only child-adult speaker classification pipeline, we propose incorporating visual cues through active speaker detection and visual processing models. Our framework involves video pre-processing, utterance-level child-adult speaker detection, and late fusion of modality-specific predictions. We demonstrate from extensive experiments that a visually aided classification pipeline enhances the accuracy and robustness of the classification. We show relative improvements of 2.38% and 3.97% in F1 macro score when one face and two faces are visible, respectively.
Introduction
In clinical child psychology, particularly for assessing neurocognitive disorders like Autism Spectrum Disorder (ASD), accurately identifying "who said what" is fundamental. Automated systems that distinguish between a child’s speech and an adult’s (e.g., a clinician or parent) enable the scaling of diagnostic tools and the calculation of critical metrics like conversational latency. However, children’s voices exhibit high variability, and clinical environments often present acoustic challenges.
Traditional speaker classification relies almost exclusively on audio. This approach faces three primary hurdles: noise in clinical recordings (often captured via consumer devices), the inherent inconsistency of child speech production, and ambiguity during rapid dyadic interactions. Visual cues, specifically lip and jaw movements, offer a direct measure of speech articulation that can disambiguate these scenarios.
Methodology
The proposed framework integrates visual information into a traditional audio-based pipeline through three stages: video pre-processing, utterance-level detection, and late fusion.
Video Pre-processing
The system first detects and tracks faces using the S3FD algorithm. Each face track is then classified as "child" or "adult" using a Vision Transformer (ViT) or ResNet model pre-trained on the FairFace dataset. This provides a probability $P_{im}(\cdot)$ that a specific face track belongs to a child.
Utterance-level Speaker Detection
The framework processes each utterance through two parallel branches:
- Audio Branch: State-of-the-art encoders (WavLM or Whisper) extract speech features to predict the speaker identity.
- Visual Branch: An adapted version of Light-ASD, a lightweight Active Speaker Detection model, is used. While the original model operates frame-by-frame, it was modified with a unidirectional Gated Recurrent Unit (GRU) to provide a single prediction for the entire utterance duration.
Late Fusion
The final classification is achieved through late fusion, combining probabilities from the audio-only classifier ($P_a$) and the visually-guided active speaker detector ($P_{asd}$) to produce the final child/adult label.
Experimental Results
The framework was evaluated on a dataset of 87 Zoom recordings of dyadic interactions between children (ages 4–8) and their parents, totaling approximately 30,000 annotated utterances.
Face Classification Performance
The effectiveness of the visual branch depends on accurate age-based face classification. Figure 5 illustrates the probability distributions for the ViT and ResNet models, showing strong separation between child and adult face tracks.
Classification Accuracy
The multimodal pipeline consistently outperformed the audio-only baseline (Whisper-Small). Key results include:
- Relative Improvements: A 2.38% gain in F1 macro score when one face was visible, increasing to 3.97% when both faces were visible.
- Peak Performance: The Whisper-Small + ResNet configuration achieved an 89.1% F1 macro score in the "2-Face" condition, compared to 85.7% for the audio-only model.
- Utterance Length: Visual cues were particularly beneficial for short utterances (0.3–0.6 seconds), where audio data is often insufficient for confident classification.
Discussion and Conclusion
Incorporating visual signals significantly enhances the robustness of child-adult speaker classification. The "Two-Face" advantage suggests that the model benefits from comparing articulation dynamics between participants. However, challenges remain in classifying non-verbal vocalizations (e.g., laughter), which lack the rhythmic lip-sync patterns typical of speech. Future work will focus on end-to-end integration of Voice Activity Detection (VAD) and improving accuracy for non-verbal cues to support more holistic behavioral assessments.
Summary updated