Leveraging pre-trained large language models for aphasia detection in English and Chinese speakers

Yan Cong, Jiyeon Lee, Arianna LaCroixNAACL 2024

TL;DR

This paper explores how pre-trained Large Language Models (LLMs) can detect aphasia, a language disorder, in both English and Mandarin Chinese speakers. The study finds that while LLMs show promise, their effectiveness varies significantly across languages and tasks, highlighting the need for linguistically tailored approaches in clinical NLP.

What Problem Does This Paper Address?

Aphasia, an acquired language disorder often caused by stroke, significantly impacts communication. While Natural Language Processing (NLP) has been applied to aphasia detection, most research focuses on English speakers. There's a significant gap in understanding how to leverage NLP, particularly advanced Large Language Models (LLMs), for non-English speakers, especially those with complex linguistic structures like Mandarin Chinese. This paper aims to bridge this gap by investigating the utility of pre-trained LLMs for aphasia detection, subtyping, and severity assessment in both English and Chinese speakers, without requiring extensive fine-tuning.

Methods

The core of this research involves using pre-trained LLMs to generate "surprisal" scores for utterances. Surprisal, a measure of how unexpected a word or sequence is given its context, is hypothesized to correlate with linguistic impairments in aphasia. The researchers used three pre-trained LLMs: GPT-2, Llama2, and BERT (specifically bert-base-chinese for Chinese and bert-base-uncased for English). For each utterance, token-wise surprisals were calculated, summed, and then averaged by the number of tokens to get a mean utterance-level surprisal. These mean surprisal scores, along with utterance length, were then used as features in logistic regression classifiers (for presence and subtype detection) and elastic net regression (for severity prediction). The study deliberately avoided fine-grained hyperparameter tuning to assess the "off-the-shelf" utility of LLMs.

Datasets

The study utilized datasets from the AphasiaBank¹ (MacWhinney et al., 2011). Participants were monolingual English or Mandarin Chinese speakers with a Western Aphasia Battery-Aphasia Quotient (WAB-AQ) of 92 or lower. For the Chinese dataset, optimal pair matching was used to match aphasia patients with control groups based on age, education, and sex, resulting in 1756 observations per group. For the English dataset, a similar matching procedure was employed, yielding 1586 observations per group. For aphasia subtype detection (Broca's and anomic aphasia), a balanced sample of 86 observations per subtype was randomly selected for both languages.

Key Insights and Ideas

  • Cross-linguistic differences matter: The performance of LLMs varied dramatically between English and Chinese speakers, suggesting that language-specific characteristics significantly influence how LLMs detect aphasia.
  • Task-dependent LLM strengths: Chinese LLMs were better at identifying aphasia subtypes (Broca's vs. anomic) than detecting the presence of aphasia. Conversely, English LLMs performed better at detecting the presence of aphasia than its subtypes.
  • Character-based processing in Chinese LLMs: The paper speculates that the character-based tokenization common in Chinese LLMs might make them more sensitive to word retrieval difficulties, aiding in subtype detection.
  • Potential impact of linguistic features: The absence of rich verb conjugations in Chinese might contribute to lower efficacy in detecting aphasia overall, as verb problems are a hallmark of the disorder in languages like English.
  • Scaling might improve performance: Larger LLMs, like Llama2, generally showed stronger coefficients in predicting aphasia severity, suggesting that model size could be a factor in clinical efficacy.

Key Results

  • Chinese Aphasia Subtyping: Logistic regression classifiers using LLM surprisals achieved an F1-score of 0.86 for aphasia subtype detection in Chinese speakers.
  • Chinese Aphasia Presence: The F1-score for detecting the presence of aphasia in Chinese speakers was 0.61.
  • English Aphasia Presence: Logistic regression classifiers achieved an F1-score of 0.79 for detecting the presence of aphasia in English speakers.
  • English Aphasia Subtyping: The F1-score for aphasia subtype detection in English speakers was 0.54.
  • Aphasia Severity Prediction: For Chinese speakers, the elastic net regression model showed better performance (lower MAE) compared to English speakers. Llama2 consistently showed the strongest coefficients across both languages for severity prediction.

Limitations and Future Work

The authors acknowledge the small sample size as a primary limitation, which restricts the study to a proof of concept rather than end-to-end predictive models. They also note that the analysis of severity prediction was based on raw data, irrespective of initial classification accuracy, which could introduce noise.

Future work suggested includes:

  • Fine-tuning LLMs: Adapting LLMs with aphasia-specific datasets to improve their performance.
  • Expanding datasets: Investigating the impact of larger datasets on LLM performance.
  • Exploring linguistic properties: Delving deeper into why LLMs perform differently across languages, examining specific linguistic features like argument structure and word order.
  • Investigating cross-linguistic similarities: Examining how shared impairment patterns in aphasia across languages can be detected using cross-linguistic LLMs.
  • Testing more LLMs: Evaluating a broader range of pre-trained LLMs.

Summary updated

BibTeX
@inproceedings{cong2024leveraging,
  author = {Yan Cong and Jiyeon Lee and Arianna LaCroix},
  title = {{Leveraging pre-trained large language models for aphasia detection in English and Chinese speakers}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2024},
}
Figure