A Cascaded Multimodal Framework for Automatic Social Communication Severity Assessment in Children with Autism Spectrum Disorder

Jihyun Mun, Sunhee Kim, Minhwa ChungInterspeech 2025

TL;DR

This paper introduces a novel, fully automated framework that uses speech and text to assess the severity of social communication challenges in children with Autism Spectrum Disorder (ASD). By combining advanced speech processing with automatic transcription and a co-attention mechanism, the system achieves a strong correlation with human expert ratings, offering a scalable and objective alternative to traditional assessment methods.

What Problem Does This Paper Address?

Assessing the severity of social communication deficits in children with Autism Spectrum Disorder (ASD) is crucial for timely intervention. However, current methods heavily rely on subjective behavioral observations and standardized tests, which can be time-consuming, prone to examiner bias, and lack objectivity. ASD significantly impacts both language use and speech production, including atypical prosody (intonation, pitch) and segmental features (pronunciation). While computational approaches have explored speech and language for ASD diagnosis, there's a need for fully automated systems that can quantify severity by jointly leveraging both acoustic and linguistic information without requiring manual transcripts or handcrafted features.

Methods

The proposed framework employs a cascaded multimodal approach, processing raw audio through several stages:

  • Speech and Text Embedding Extraction:
    • Speech: Two pre-trained foundation models are used:
  • TRILLsson: For suprasegmental features (prosody, intonation).
  • wav2vec2.0-xlsr: For segmental features (phonetics, pronunciation). These models are combined using a transformer-like block, where wav2vec2.0 embeddings act as queries and TRILLsson embeddings as keys/values, enriching segmental information with prosodic context.

  • Text: Since manual transcription is labor-intensive, an Automatic Speech Recognition (ASR) model (Whisper-large-v2) is fine-tuned on ASD speech data to generate transcriptions. These transcriptions, along with manually transcribed text, are then processed by a pre-trained Korean language model (klue/RoBERTa-base) to extract text embeddings.

  • Co-attention for Multimodal Fusion: A co-attention mechanism is used to fuse speech and text embeddings. This involves two parallel processes:
  • Speech-conditioned text representation: Text embeddings are queried by speech embeddings.
  • Text-conditioned speech representation: Speech embeddings are queried by text embeddings. This bidirectional attention allows each modality to inform the other, creating richer, cross-modal representations.

  • Feature Fusion and Prediction: The fused representations are concatenated and passed through a fully connected layer to a regression head, which predicts the ASD severity level. The framework uses a regression approach to capture the ordinal nature of severity.

Datasets

The study utilizes the Korean ASD corpus introduced by [7]. This corpus contains speech data from children with ASD, with severity independently assessed by three speech-language pathologists, and an average score used as the ground truth. The dataset is split into training, validation, and test sets in an 8:1:1 ratio, ensuring speaker independence and stratified severity distribution.

Key Insights and Ideas

  • Multimodality is Key: Combining speech (both segmental and suprasegmental) and text information significantly outperforms unimodal approaches for ASD severity assessment.
  • ASR's Nuanced Role: While ASR-generated text alone can be a better predictor than manual transcripts in some contexts (preserving subtle articulatory details), its benefit in multimodal settings depends on the richness of the speech features. In the full model, ASR text slightly outperforms manual transcripts, suggesting that a comprehensive speech representation can better leverage the detailed information in ASR outputs.
  • Suprasegmental Cues Need Context: TRILLsson (suprasegmental) embeddings alone perform poorly, indicating that prosodic features require integration with segmental or lexical information for effective severity estimation.
  • wav2vec2.0's Strength: The wav2vec2.0 model alone shows strong predictive power, highlighting the importance of segmental speech features in ASD.
  • Scalability through Automation: The framework's ability to process raw audio and use ASR for transcription eliminates the need for manual feature engineering and human annotation, making it highly scalable and applicable in real-world scenarios.

Key Results

The proposed fully automated multimodal framework (Configuration 11) achieved the highest Spearman's correlation coefficient (SCC) of 0.5629 with human ratings for ASD severity assessment. This significantly outperforms unimodal configurations and previous multimodal approaches that relied on manual features or transcripts. For comparison:

  • wav2vec2.0 alone: SCC of 0.4971
  • TRILLsson alone: SCC of 0.1235
  • Manual text alone: SCC of 0.1452
  • ASR text alone: SCC of 0.1831

Limitations and Future Work

  • Limitations: The authors acknowledge that their SCC of 0.5629 is slightly lower than a previous study (0.567) that used handcrafted features and manual transcripts. However, they emphasize that their approach is fully automated and more scalable. The performance of ASR text in some multimodal configurations was slightly lower than manual transcripts, suggesting potential noise from ASR errors that can interfere with single speech modalities.
  • Future Work: The authors suggest exploring more advanced speech or language models and alternative fusion strategies. They also propose extending the research to jointly diagnose ASD and estimate severity, and investigating multitask learning to assess other speech-related dimensions like pronunciation proficiency.

Summary updated

BibTeX
@inproceedings{mun2025cascaded,
  author = {Jihyun Mun and Sunhee Kim and Minhwa Chung},
  title = {{A Cascaded Multimodal Framework for Automatic Social Communication Severity Assessment in Children with Autism Spectrum Disorder}},
  booktitle = {Interspeech},
  year = {2025},
}
Figure