Audio-Visual Child-Adult Speaker Classification in Dyadic Interactions

Anfeng Xu, Kevin Huang, Tiantian Feng, Helen Tager-Flusberg, Shrikanth Narayanan

In the world of clinical child psychology—specifically for assessing Autism Spectrum Disorder (ASD)—the "who said what" of a conversation is just as important as the "what." Automated systems that can distinguish between a child’s speech and an adult’s (usually a clinician or parent) are vital for scaling diagnostic tools. However, children’s voices are notoriously variable, and clinical environments are rarely studio-quiet.

A new paper from researchers at the University of Southern California and Boston University proposes a solution: stop just listening and start looking. By integrating visual lip-motion data with traditional audio processing, they’ve created a more robust multimodal pipeline for child-adult speaker classification.

TL;DR

Researchers have developed a multimodal framework that combines audio and visual signals to distinguish between child and adult speakers in dyadic (one-on-one) interactions. By leveraging active speaker detection and age-based face classification, the system achieves a nearly 4% improvement in F1 macro scores over traditional audio-only methods, particularly in complex visual environments.

What Problem Does This Paper Address?

Automated analysis of child-adult interactions is a cornerstone for modern neurocognitive disorder assessments. To calculate metrics like "conversational latency" or "intelligible child speech," a computer first needs to accurately segment the audio into child vs. adult speech.

Traditional systems rely almost exclusively on audio. This is problematic because:

  1. Audio Quality: Clinical recordings (often via Zoom or handheld devices) can be noisy.
  2. Speech Variability: Children’s speech production is highly inconsistent compared to adults.
  3. Ambiguity: In a fast-paced dyadic interaction, audio-only models often struggle to distinguish between overlapping voices or similar pitch ranges.

The authors argue that visual cues—specifically the movement of the lips and jaw—provide a direct, unambiguous measure of who is actually speaking at any given moment.

Methods

The proposed framework consists of three main stages:

  1. Video Pre-processing: The system uses the S3FD algorithm to detect and track faces throughout the video. It then employs a Vision Transformer (ViT) or ResNet (pre-trained on the FairFace dataset) to classify each face track as either "child" or "adult."
  2. Utterance-level Speaker Detection:
    • Audio: The team used state-of-the-art pre-trained encoders like WavLM and Whisper to extract speech features.
  3. Visual: They adapted Light-ASD, a lightweight Active Speaker Detection model. While the original Light-ASD works frame-by-frame, the authors modified it to predict the speaker for an entire utterance using a unidirectional Gated Recurrent Unit (GRU).
  4. Late Fusion: The final classification isn't just a guess from one model. The system performs "late fusion," combining the probabilities from the audio-only classifier and the visually-guided active speaker detector to make a final decision.

Datasets

  • Primary Dataset: 87 Zoom recordings of dyadic activities between children (ages 4–8) and their parents. This provided approximately 30,000 annotated utterances.
  • Pre-training Benchmarks: The visual models were pre-trained on FairFace (for age classification) and AVA-Active Speaker (for detecting if a person is talking).
  • Language Levels: The dataset included children at various developmental stages, from pre-verbal communication to full word combinations.

Key Insights and Ideas

  • The "Two-Face" Advantage: The system performs best when both the child and adult are visible. In this "2-Face" condition, the model can compare the articulation dynamics of both participants simultaneously, effectively using one as a baseline for the other.
  • Visuals as a Stabilizer: Visual cues are most beneficial for shorter utterances (0.3–0.6 seconds). Audio models often lack enough data in a half-second clip to be certain, but a clear visual of a child’s mouth moving provides the necessary "tie-breaker."
  • The Non-Verbal Hurdle: Interestingly, the model still struggles with non-verbal vocalizations (laughing, crying, etc.). This is because these sounds often lack the specific, rhythmic lip-sync patterns that Active Speaker Detection models are trained to recognize in uttered speech.

Key Results

The researchers compared their multimodal approach against a baseline audio-only model (using Whisper-Small):

  • Overall Improvement: The multimodal pipeline showed a relative improvement of 2.38% in F1 macro score when one face was visible and 3.97% when two faces were visible.
  • Top Performance: The best configuration (Whisper-Small + ResNet) achieved an 89.1% F1 macro score in the 2-Face condition, compared to 85.7% for audio-only.
  • Robustness: The multimodal approach consistently outperformed audio-only across all utterance lengths, with the most significant gains seen in the shortest segments.

Limitations and Future Work

While the results are promising, the authors noted a few areas for growth:

  • Non-Dyadic Scenarios: The current model is optimized for one-on-one interactions. Real-world clinical settings might involve multiple clinicians or siblings, requiring a more complex multi-speaker tracking system.
  • VAD Integration: The current pipeline assumes the speech segments are already identified. Integrating Voice Activity Detection (VAD) directly into the multimodal framework could make the system fully end-to-end.
  • Non-Verbal Accuracy: Improving the detection of non-verbal cues is essential for a holistic understanding of child behavior in ASD assessments.

Summary updated

Abstract

Interactions involving children span a wide range of important domains from learning to clinical diagnostic and therapeutic contexts. Automated analyses of such interactions are motivated by the need to seek accurate insights and offer scale and robustness across diverse and wide-ranging conditions. Identifying the speech segments belonging to the child is a critical step in such modeling. Conventional child-adult speaker classification typically relies on audio modeling approaches, overlooking visual signals that convey speech articulation information, such as lip motion. Building on the foundation of an audio-only child-adult speaker classification pipeline, we propose incorporating visual cues through active speaker detection and visual processing models. Our framework involves video preprocessing, utterance-level child-adult speaker detection, and late fusion of modality-specific predictions. We demonstrate from extensive experiments that a visually aided classification pipeline enhances the accuracy and robustness of the classification. We show relative improvements of 2.38% and 3.97% in F1 macro score when one face and two faces are visible, respectively.
BibTeX
@inproceedings{xu2024audio,
  author = {Anfeng Xu and Kevin Huang and Tiantian Feng and Helen Tager-Flusberg and Shrikanth Narayanan},
  title = {{Audio-Visual Child-Adult Speaker Classification in Dyadic Interactions}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2024},
  url = {https://openreview.net/forum?id=ESypQE4ki4},
}

Loading related content...

Figure