Using natural conversations to classify autism with limited data: Age matters

Michael Hauser, Evangelos Sariyanidi, Birkan Tunc, Casey Zampella, Edward Brodkin, Robert Schultz, Julia Parish-MorrisNAACL 2019

TL;DR

This paper tackles the challenge of classifying Autism Spectrum Disorder (ASD) using natural conversation data, which is complicated by significant individual variability and limited large datasets. By incorporating developmental knowledge about language use across childhood, adolescence, and adulthood, the researchers developed age-specific machine learning models that significantly outperform a single, age-agnostic model, demonstrating the critical role of age in understanding ASD-related communication differences.

What Problem Does This Paper Address?

Autism Spectrum Disorder (ASD) is characterized by social communication impairments. While spoken language is a key area for diagnosis and understanding, it's highly variable in individuals with ASD, ranging from severely impaired to verbally gifted. This heterogeneity, coupled with the scarcity of large, well-annotated datasets for psychiatric conditions, makes it difficult to develop robust machine learning models for classification and characterization. Traditional machine learning often assumes data homogeneity and benefits from large datasets, both of which are problematic for ASD research. This paper addresses the need for more accurate and reliable ASD classification by exploring how to leverage domain knowledge, specifically developmental changes in language, to overcome data limitations and individual variability.

Methods

The study employed two machine learning approaches to classify individuals with ASD versus typically developing (TD) controls using naturalistic conversation samples.

  • Age-Agnostic Model: A single model was trained to classify all participants regardless of age.
  • Age-Specific Models: Participants were divided into three age groups: middle childhood (8-11 years), adolescence (12-17 years), and adulthood (18-50 years). Separate models were then trained for each age group.

The core of the methodology involved extracting a rich set of linguistic features from audio recordings. These features included:

  • Dyadic Features: Derived from the interaction between the participant and their conversational partner (confederate), such as turn-taking rates.
  • Interlocutor Features: Characteristics of the confederate's speech.
  • Participant Features: Linguistic patterns of the individual with ASD.
  • Difference Metrics: Calculated by comparing features within a speaker across different conversational conditions (e.g., "interested" vs. "bored" confederate) or between speakers within a condition. This approach was designed to guide the model towards learning clinically relevant dependencies.

A logistic regression model (specifically, Maximum Entropy classifier) was used for classification. Features were down-selected based on their f-value (indicating separation between groups), and an l2-regularization penalty was applied. The models were evaluated using a leave-one-out cross-validation strategy.

Datasets

The study utilized a dataset of 140 participants: 81 with ASD and 59 typically developing (TD) controls. These participants were matched for age and IQ. The dataset was further segmented into three age groups:

  • Middle Childhood: 8-11 years
  • Adolescence: 12-17 years
  • Adulthood: 18-50 years

The data consisted of 6-minute naturalistic conversation samples. The conversations were recorded using a specialized "TreeCam" device that captured audio and video from both participants and their confederates. The confederates were trained research staff who engaged participants in two conditions: "Interested" and "Bored."

Key Insights and Ideas

  • Age is a Critical Factor: The most significant insight is that accounting for developmental stage dramatically improves classification accuracy. This challenges the conventional machine learning wisdom that more data is always better, suggesting that how data is structured (by age, in this case) can be more impactful than sheer volume when dealing with heterogeneous populations.
  • Linguistic Features Evolve: The specific linguistic markers that best distinguish ASD from TD are not static but change significantly across developmental stages. This implies that a "one-size-fits-all" approach to linguistic analysis for ASD is insufficient.
  • Domain Knowledge is Powerful: Incorporating prior knowledge from developmental psychology and clinical observation (e.g., how language use changes with age) can guide machine learning models to learn more relevant and effective features, especially when data is limited.
  • Conversational Context Matters: The study highlights the importance of analyzing natural conversations, including dyadic and interlocutor features, rather than relying solely on decontextualized language samples. The "Bored" condition, in particular, proved informative for adolescents.
  • Heterogeneity is Manageable: While ASD is known for its heterogeneity, this study demonstrates that by segmenting the population based on developmental stage, this heterogeneity can be managed to achieve more reliable diagnostic models.

Key Results

  • Significant Accuracy Improvement: The age-specific models achieved a weighted average classification accuracy of 0.829, a 20.8% relative increase compared to the age-agnostic model, which achieved 0.686 accuracy (p < 0.001).
  • Age Group Performance: Individual age-specific models showed strong performance:
  • 8-11 years: 0.756 accuracy
    • 12-17 years: 0.806 accuracy
    • 18-50 years: 0.889 accuracy
  • Feature Variation by Age:
    • Children (8-11): Pronoun use (especially first-person plural "we"), out-of-dictionary words, and speech rate were key discriminators.
  • Adolescents (12-17): Features related to the "Bored" condition, confederate authenticity, and pronoun use (second-person "you") were important.
  • Adults (18-50): Temporal features like overlapping pauses and the rate of pauses per minute were most predictive, suggesting differences in conversational flow rather than topic.

Limitations and Future Work

  • Limitations:
    • The study acknowledges that dividing the dataset into age groups inherently reduces the amount of data available for each individual model, which contrasts with the general machine learning principle of needing more data. However, they demonstrate that the gains from age-specific modeling outweigh this reduction.
  • The paper focuses on logistic regression, a linear model, which is interpretable but may not capture all complex non-linear relationships.
  • Future Work:
    • Conduct longitudinal studies to track treatment progress by monitoring feature values over time.
  • Extend the techniques to differentiate ASD from other mental health disorders (e.g., anxiety, depression, obsessive-compulsive disorder).
  • Incorporate additional features, such as pitch-related acoustic features.
  • Explore more complex nonlinear models for classification and prediction in larger sample sizes.
  • Investigate how these linguistic features can inform the development of targeted interventions.

Summary updated

BibTeX
@inproceedings{hauser2019using,
  author = {Michael Hauser and Evangelos Sariyanidi and Birkan Tunc and Casey Zampella and Edward Brodkin and Robert Schultz and Julia Parish-Morris},
  title = {{Using natural conversations to classify autism with limited data: Age matters}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2019},
}
Figure