TL;DR
Assessing child-adult interactions during clinical autism evaluations is critical but heavily labor-intensive. This paper explores whether generic, open-source Large Language Models (LLMs) can assist clinicians by analyzing clinical transcripts across four core tasks: speaker classification, activity recognition, language level assessment, and age estimation. The authors find that local LLMs (<15B parameters) not only match trained baselines on utterance classification but also outperform non-expert human evaluators on activity and language skill estimation.
What Problem Does This Paper Address?
Standard clinical assessments for Autism Spectrum Disorder (ASD)—such as the Autism Diagnostic Observation Schedule (ADOS)—rely heavily on semi-structured, dyadic interactions between a child and an adult (clinician or parent). These sessions produce lengthy, complex verbal interactions marked by frequent repetitions, abrupt topic shifts, non-verbal gestures, and atypical language use.
Manually transcribing, segmenting, and assessing these sessions requires significant expert clinical time, while non-experts struggle with cognitive fatigue and subtle conversational cues. Although LLMs excel at processing generic adult conversations, their ability to reason over messy, low-resource, child-inclusive clinical transcripts remains understudied. Due to strict institutional review board (IRB) and privacy requirements, researchers cannot send sensitive clinical dialogues to commercial cloud APIs (like GPT-4o), necessitating the evaluation of locally deployable, open-source models.
Methods
The authors benchmarked six open-source instruction-tuned LLMs under 15 billion parameters: Mistral-7B V0.2, LLaMA-2 (7B and 13B), LLaMA-3-8B, and Qwen1.5 (7B and 14B), loaded in float16 precision.
The models were evaluated on four clinically motivated tasks:
- Child-Adult Utterance Classification (Utterance-level): Classifying whether an isolated utterance was spoken by the child or the adult, evaluated in 0-shot and 5-shot conversation-context settings (benchmarked against a fine-tuned RoBERTa baseline).
- Engaged Activity / Observation Recognition (Session-level): Identifying the primary activity or observation focus across the entire dialogue transcript.
- Language Skill Evaluation (Session-level): Categorizing the child’s expressive language stage (e.g., pre-verbal / first words vs. multi-word combinations).
- Individual Trait Prediction (Session-level): Predicting the child's binarized age range based on conversational cues.
To improve LLM inference, the researchers experimented with Chain-of-Thought (CoT) prompting, majority voting ensembles across top-performing models, and direct comparisons against non-expert human annotators.
Datasets
The authors evaluated two distinct interaction paradigms:
- Remote-NLS: 89 Zoom recordings (~15 minutes each) of naturalistic parent-child play sessions designed to elicit Natural Language Samples (NLS) from mostly minimally verbal children with ASD (mean age: 75 months). Sessions were transcribed and labeled for 13 multi-label play activities and developmental language stages.
- ADOSMod3: Semi-structured clinical interactions between clinicians and 164 verbally fluent children with and without ASD (mean age: 104 months) administering ADOS Module 3. Focus was placed on two 3-minute subtasks: "Emotion" and "Social Difficulties and Annoyance".
Both datasets used human expert transcriptions rather than automated speech recognition (ASR) outputs.
Key Insights and Ideas
- LLMs beat non-expert humans at clinical language tasks: LLMs consistently outperformed non-expert human reviewers on identifying activities and estimating language levels from transcripts, highlighting their utility as automated screening or clinical assistant tools.
- Context is essential, but more isn't always better: For utterance speaker classification, adding 5 preceding dialogue turns dramatically boosted F1 scores across most models. However, some models (like LLaMA-3-8B) degraded with context, showing sensitivity to unstructured, repetitive child dialogues.
- Chain-of-Thought (CoT) prompting backfires: Asking models to generate reasoning chains prior to answering consistently degraded performance across all session-level tasks. Manual inspection revealed that reasoning prompts triggered hallucinations and repetitive gibberish rather than sound logic.
- Model ensembling delivers quick wins: Simple majority voting across the top three LLMs consistently improved session-level performance across activity, language, and demographic classification.
Key Results
- Speaker Classification: With 5 history utterances, Mistral-7B jumped from 0.530 to 0.693 F1 on ADOSMod3 and achieved 0.599 F1 on Remote-NLS, approaching the fine-tuned RoBERTa baseline (0.807 on ADOSMod3, 0.606 on Remote-NLS).
- Activity Recognition: Qwen1.5-14B reached 0.984 F1 on ADOSMod3 binary observation classification. On the harder 13-label Remote-NLS dataset, majority voting achieved 0.527 F1, comfortably beating non-expert human evaluators (0.448 vs. 0.332 F1 on a sample subset).
- Language Skill Assessment: Mistral-7B reached 0.795 F1 on Remote-NLS (boosted to 0.817 F1 via majority voting). On a randomly sampled human-evaluation subset, the top LLM achieved an F1 of 1.000, compared to just 0.592 for non-expert humans.
- Individual Traits (Age): Predicting age brackets proved difficult, particularly for younger children under 8 (Remote-NLS best F1 was 0.500 with voting, compared to 0.722 F1 on ADOSMod3).
Limitations and Future Work
- Text-Only Modality: The study evaluated textual transcripts, missing crucial non-verbal information (facial expressions, eye contact, acoustic prosody, and gestures) that clinicians rely on. Future iterations should incorporate multimodal speech and vision foundation models (VLMs).
- Clean Transcripts vs. Real-world Noise: Experiments used gold-standard expert transcriptions; real-world deployment requires evaluating robustness against noisy, automated ASR and diarization pipelines.
- Hallucination Risks & Biases: Small open LLMs frequently exhibited prompt confusion and fabricated text justifications, highlighting safety hurdles before these tools can be directly integrated into diagnostic workflows.
Summary updated