Predicting pragmatic discourse features in the language of adults with autism spectrum disorder

Christine Yang, Duanchen Liu, Qingyun Yang, Zoey Liu, Emily Prud’hommeauxACL 2021

TL;DR

This paper introduces a new dataset of natural conversations from adults with and without Autism Spectrum Disorder (ASD) and develops neural models to automatically predict pragmatic discourse features like politeness, uncertainty, and informativeness. The goal is to better understand and potentially identify communication differences in adults with ASD, with the BERT-based model showing promising results.

What Problem Does This Paper Address?

Individuals with Autism Spectrum Disorder (ASD) often experience challenges in social communication, particularly in discourse and pragmatic expression. While these difficulties are recognized, precisely quantifying them, especially in adults, remains a challenge. Existing methods rely heavily on time-consuming manual annotation of language, and computational approaches have largely focused on children and semi-structured interactions, not spontaneous peer conversations. This paper aims to bridge this gap by creating a novel dataset and developing automated methods to identify pragmatic features that can distinguish between adults with and without ASD.

Methods

The researchers collected natural conversations from adults with ASD and neurotypical (TD) individuals engaged in collaborative tasks. These conversations were manually transcribed and annotated for three pragmatic features: politeness, uncertainty, and informativeness, each on a 1-3 scale. To automate this process, they trained several machine learning models. The core of their approach involves using pre-trained language embeddings, specifically BERT embeddings, within a feedforward neural network. They also experimented with LSTM and CNN models using word embeddings (word2vec and GloVe). The models were trained to predict the manually annotated categorical labels for each feature.

Datasets

The primary dataset introduced in this paper consists of transcribed natural conversations from 14 adults with high-functioning ASD and 8 adults with typical development (TD), each paired with a neurotypical conversational partner. This dataset comprises 9,267 utterances from experimental participants. The conversations were collected during collaborative tasks, specifically a map task and a deserted island task. This dataset is notable for being one of the few available for spontaneous adult conversations with peers and the first to be manually annotated for all three pragmatic features (politeness, uncertainty, informativeness).

Key Insights and Ideas

  • Adult ASD communication is nuanced and task-dependent: The study found significant differences in pragmatic features between ASD and TD groups, but these differences varied depending on the task. For instance, ASD participants were more polite in the map task (instruction-giving) but less polite in the island task (collaborative decision-making). This highlights that communication styles are not fixed but context-dependent.
  • Automated prediction is feasible and promising: The development of neural models, particularly those leveraging BERT embeddings, shows that pragmatic features can be predicted with high accuracy, often outperforming previous approaches. This opens doors for more efficient and scalable analysis.
  • BERT excels in capturing pragmatic nuances: The BERT-based feedforward neural network consistently achieved the highest F1 scores across all three pragmatic features, demonstrating its effectiveness in understanding the subtle linguistic cues associated with politeness, uncertainty, and informativeness.

Key Results

The manual annotations revealed significant differences between ASD and TD participants in politeness and informativeness in the map task, and in uncertainty and informativeness in the island task. The BERT-based feedforward neural network model achieved the highest performance, with F1 scores exceeding 0.82 for all three pragmatic features. Specifically, the BERT model outperformed the majority baseline by 0.1 for politeness, 0.33 for uncertainty, and 0.42 for informativeness. However, the models did not fully capture the statistically significant group differences observed in the manual annotations, showing some reversed tendencies in certain features and tasks.

Limitations and Future Work

The authors acknowledge that while their models achieve high accuracy, they do not fully capture the statistically significant differences in pragmatic expression observed in the manual annotations. There is still work to be done in fine-tuning the models to better reflect these between-group differences. Future work includes exploring different model architectures, leveraging external labeled corpora for more systematic comparisons, expanding the dataset with more participants (aiming for 20 ASD and 20 TD), and incorporating annotations for other pragmatic features like coherence and dialog acts to gain a more comprehensive understanding of pragmatic expression in adults with ASD.

Summary updated

BibTeX
@inproceedings{yang2021predicting,
  author = {Christine Yang and Duanchen Liu and Qingyun Yang and Zoey Liu and Emily Prud’hommeaux},
  title = {{Predicting pragmatic discourse features in the language of adults with autism spectrum disorder}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
}
Figure