Can Generic LLMs Help Analyze Child-adult Interactions Involving Children with Autism in Clinical Observation?

Tiantian Feng, Anfeng Xu, Rimita Lahiri, Helen Tager-Flusberg, So Hyun Kim, Somer Bishop, Catherine Lord, Shrikanth Narayanan

Abstract

Large Language Models (LLMs) have shown significant potential in understanding human communication and interaction. However, their performance in the domain of child-inclusive interactions, including in clinical settings, remains less explored. In this work, we evaluate generic LLMs' ability to analyze child-adult dyadic interactions in a clinically relevant context involving children with ASD. Specifically, we explore LLMs in performing four tasks: classifying child-adult utterances, predicting engaged activities, recognizing language skills and understanding traits that are clinically relevant. Our evaluation shows that generic LLMs are highly capable of analyzing long and complex conversations in clinical observation sessions, often surpassing the performance of non-expert human evaluators. The results show their potential to segment interactions of interest, assist in language skills evaluation, identify engaged activities, and offer clinical-relevant context for assessments.

Introduction

Clinical assessments for Autism Spectrum Disorder (ASD), such as the Autism Diagnostic Observation Schedule (ADOS), rely heavily on semi-structured dyadic interactions between a clinician and a child. Analyzing these sessions is a labor-intensive process that requires significant clinical expertise. While Large Language Models (LLMs) have excelled in general human communication tasks, "child-inclusive" interactions present a unique challenge. These conversations are often non-linear, filled with repetitions, pauses, and idiosyncratic language patterns that deviate from standard adult discourse.

This research evaluates whether open-source, generic LLMs can automate the "heavy lifting" of transcript analysis. By focusing on speaker diarization, activity segmenting, and developmental trait prediction, the study aims to determine if these models can effectively assist clinicians in diagnostic workflows by providing structured insights from raw transcripts.

Methodology

The researchers benchmarked several locally deployable, open-source LLMs to ensure data privacy and compliance with institutional review boards. The models tested included Mistral-7B, LLaMa-2 (7B/13B), LLaMa-3 (8B), and Qwen-1.5 (7B/14B).

Evaluation was conducted across four primary tasks:

  1. Child-adult Utterance Classification: Determining if a specific line was spoken by the child or the adult.
  2. Activity/Observation Recognition: Identifying which clinical tasks (e.g., "Coloring art" or "Emotion discussion") occurred during the session.
  3. Language Skill Prediction: Categorizing the child's linguistic development level (e.g., pre-verbal vs. word combinations).
  4. Trait Prediction: Predicting the child’s age range based on conversational cues.
Example of the activity recognition task where the LLM identifies the primary activity from a conversation snippet.

The team employed various prompting strategies, including 0-shot and 5-shot learning. They also explored Chain-of-Thought (CoT) reasoning and Majority Voting across the top-performing models to improve reliability.

Datasets

The study utilized two distinct datasets representing different clinical contexts and levels of verbal fluency:

  • Remote-NLS: This dataset consists of 89 naturalistic interactions recorded via Zoom at home. These sessions involve parents and children who are often minimally verbal, providing a challenging, less structured environment.
  • ADOSMod3: This dataset includes 164 semi-structured clinical sessions following the ADOS-2 protocol. These sessions focus on children who are more verbally fluent and follow a more predictable clinical structure.
Summary of dataset statistics and examples of language skill prompts used for evaluation.

Both datasets utilized high-quality transcripts annotated by human experts to serve as the ground truth for model evaluation.

Key Results and Insights

The evaluation revealed several critical findings regarding the performance of LLMs in clinical settings:

  • Context is Essential for Diarization: In 0-shot settings, LLMs struggled to distinguish between child and adult utterances. However, providing just five previous lines of dialogue (5-shot) significantly boosted performance. This suggests that LLMs rely on "conversational flow" to recognize the power dynamics and linguistic complexity differences between a clinician and a child.
  • LLMs vs. Non-Expert Humans: In session-level tasks like activity recognition and language skill assessment, LLMs consistently outperformed non-expert human evaluators. While humans often suffered from fatigue or missed subtle cues in long transcripts, LLMs maintained consistent attention across the entire session. For example, on the Remote-NLS dataset, the best LLM achieved an F1-score of 0.795, vastly outperforming non-expert humans who averaged 0.592.
  • The "CoT" Penalty: Interestingly, Chain-of-Thought (CoT) prompting—which usually enhances LLM performance—actually decreased accuracy in this context. The models tended to "hallucinate" evidence to justify their reasoning, leading to incorrect final classifications.
  • Structure and Performance: Models performed significantly better on the ADOSMod3 dataset than the Remote-NLS. Qwen1.5-14B was exceptionally strong in activity recognition, reaching an F1-score of 0.984 on the ADOS dataset. This suggests that more structured clinical protocols are easier for generic LLMs to parse.

Limitations and Future Directions

Despite the promising results, several hurdles remain before these tools can be deployed in clinical environments. A primary concern is hallucination, where LLMs occasionally generate random words or fabricate clinical justifications. Furthermore, this study relied on expert-level manual transcripts; real-world applications would need to handle the errors inherent in Automated Speech Recognition (ASR).

Additionally, ASD diagnosis relies heavily on non-verbal cues such as eye contact, gestures, and tone of voice, which text-only LLMs cannot currently process. Future research will explore Vision Language Models (VLMs) and speech foundation models to incorporate multi-modal data (video and audio) into the diagnostic analysis.

Summary updated

Abstract

Large Language Models (LLMs) have shown significant potential in understanding human communication and interaction. However, their performance in the domain of child-inclusive interactions, including in clinical settings, remains less explored. In this work, we evaluate generic LLMs' ability to analyze child-adult dyadic interactions in a clinically relevant context involving children with ASD. Specifically, we explore LLMs in performing four tasks: classifying child-adult utterances, predicting engaged activities, recognizing language skills and understanding traits that are clinically relevant. Our evaluation shows that generic LLMs are highly capable of analyzing long and complex conversations in clinical observation sessions, often surpassing the performance of non-expert human evaluators. The results show their potential to segment interactions of interest, assist in language skills evaluation, identify engaged activities, and offer clinical-relevant context for assessments.
BibTeX
@inproceedings{feng2024can,
  author = {Tiantian Feng and Anfeng Xu and Rimita Lahiri and Helen Tager-Flusberg and So Hyun Kim and Somer Bishop and Catherine Lord and Shrikanth Narayanan},
  title = {{Can Generic LLMs Help Analyze Child-adult Interactions Involving Children with Autism in Clinical Observation?}},
  booktitle = {GenAI4Health @ NeurIPS},
  year = {2024},
  url = {https://openreview.net/forum?id=3Pynfd5HIy},
}
Figure