Distributional semantic models for the evaluation of disordered language

Masoud Rouhizadeh, Emily Prud’hommeaux, Brian Roark, Jan van SantenNAACL 2013

TL;DR

This paper introduces a computational method using distributional semantic models to automatically identify "unexpected" words in the language of children with autism. This approach shows promise for a more objective and scalable way to assess language atypicalities, potentially aiding in the diagnosis of Autism Spectrum Disorder (ASD).

What Problem Does This Paper Address?

Autism Spectrum Disorder (ASD) is characterized by impaired communication, and atypical language, particularly in semantics and pragmatics, is a core symptom. While clinical instruments often rely on subjective parental reports or general impressions to assess these language differences, there's a need for more objective and quantifiable measures. Specifically, the rate at which children with ASD use unusual or unexpected words and phrases in their spontaneous language is not well-measured. This paper aims to bridge this gap by developing automated methods to identify and quantify these semantically unexpected words, offering a potential tool for clinical evaluation.

Methods

The core of the paper lies in using distributional semantic models to identify unexpected words. The process involves several key steps:

  • Narrative Retelling: Participants (children with ASD and typically developing peers) listen to a short story and then retell it. This provides a controlled yet spontaneous language sample.
  • Word Association Measures: Two standard information retrieval techniques, TF-IDF (Term Frequency-Inverse Document Frequency) and log-odds ratio, are employed. These measures score words based on their specificity to a particular retelling relative to the entire corpus of retellings. Words with high scores are considered potentially unique or unexpected within that specific narrative context.
  • Thresholding: A leave-one-out cross-validation approach is used to determine an optimal threshold for each word association measure. Words scoring above this threshold are flagged as potentially unexpected.
  • Filtering with External Resources: To refine the set of unexpected words and reduce false positives, two filtering steps are applied:
  • WordNet Synonyms: The WordNet lexical database is used to check if synonyms of a flagged word appear in the source narrative or adult retellings. If a synonym is present, the word is considered less likely to be truly unexpected.
  • CHILDES Corpus Analysis: The CHILDES corpus of conversational speech is used to estimate the topic associated with a flagged word. If more than two words representing that topic are also present in the source narrative or adult retellings, the flagged word is eliminated.
  • Evaluation: The performance of these automated methods is evaluated against manually annotated data, comparing precision, recall, and F-measure.

Datasets

  • Narrative Retellings: The primary dataset consists of narrative retellings from:
  • 37 typically developing children (TD)
    • 21 children with Autism Spectrum Disorder (ASD)
    • Neurotypical adults (collected via Amazon Mechanical Turk)
  • External Resources:
    • WordNet: Used for semantic expansion and synonym checking.
  • CHILDES Corpus: Used for topic estimation and filtering.

Key Insights and Ideas

  • Quantifying Atypicality: The paper demonstrates that computational methods can effectively quantify a specific type of language atypicality (unexpected word use) in children with ASD.
  • Distributional Semantics as a Diagnostic Tool: The success of distributional semantic models in identifying these subtle linguistic differences suggests their potential as valuable tools in clinical language assessment.
  • Filtering is Crucial: The filtering steps, particularly using WordNet, significantly improve the accuracy of unexpected word identification, highlighting the importance of contextual and semantic validation.
  • ASD vs. TD Differences: The study confirms that children with ASD do indeed produce a significantly higher rate of semantically unexpected words compared to their typically developing peers.

Key Results

  • The automated methods, especially when combined with filtering, significantly outperform a simple baseline method (identifying words never used in other retellings) in identifying unexpected words.
  • The best-performing method (Max(TF-IDF, Log-odds)+WordNet) achieved an F1 score of 83.9.
  • Manually annotated data showed that children with ASD had approximately 7.5% of their words marked as unexpected, compared to 2.5% in the TD group, a significant difference (p < 0.01).
  • The automated methods replicated this significant difference, with the best method estimating 6.6% unexpected words in the ASD group and 2.5% in the TD group (p < 0.01).

Limitations and Future Work

  • Limitations:
    • The authors acknowledge that the rate of unexpected word use alone might not be sufficient to classify diagnostic groups.
  • The current approach focuses on lexical-level unexpectedness and may not capture all forms of semantic or pragmatic atypicality.
  • Future Work:
    • Optimizing the number of topical words selected during filtering to balance precision and recall.
  • Incorporating part-of-speech, word sense, and parse information to enhance semantic expansion and topic estimation.
  • Using unexpected word use as one feature among many to capture the broader range of semantic and pragmatic atypicalities in ASD.
  • Applying these methods to identify confabulations and topic shifts in the elderly with neurodegenerative conditions.

Summary updated

BibTeX
@inproceedings{rouhizadeh2013distributional,
  author = {Masoud Rouhizadeh and Emily Prud’hommeaux and Brian Roark and Jan van Santen},
  title = {{Distributional semantic models for the evaluation of disordered language}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2013},
}
Figure