Intended Target Identification for Anomia Patients with Gradient-based Selective Augmentation

Jongho Kim, Romain Storaï, Seung-won Hwang

Introduction

Anomia, a hallmark symptom of aphasia, involves a specific deficit in word retrieval where individuals know their intended target but cannot produce the word. To compensate, patients often employ circumlocution, describing the target concept through related terms. For instance, a patient might say "the woman who uses magic for the girl" instead of "Fairy Godmother."

Automated identification of these targets faces two primary linguistic hurdles:

  1. Term Failure (C1): The most descriptive keywords are often omitted from the patient's speech.
  2. Semantic Paraphasia (C2): Patients frequently substitute the target with related but incorrect words (e.g., "queen" instead of "godmother"), introducing misleading "noise" for standard retrieval algorithms.
Linguistic challenges in anomia: Term failure (C1) and semantic paraphasia (C2).

Methodology: The GradSelect Framework

To address these challenges, researchers developed GradSelect, a two-stage gradient-based data augmentation strategy. GradSelect leverages the internal signals of a model—specifically its gradients—to distinguish between helpful descriptions and misleading errors.

Denoising via Gradient Magnitude

GradSelect identifies tokens that heavily influence the model's predictions by calculating their gradient magnitude. The authors observed that "unperturbed" keywords (the reliable parts of a description) typically exhibit the highest gradients. By selectively injecting noise into other high-impact terms while shielding the top-tier keywords, the model is forced to learn from the most semantically robust anchors.

Expansion via Gradient Variance

To mitigate the absence of key terms (C1), GradSelect uses gradient variance to identify relevant but "unseen" terms. This approach, inspired by pseudo-relevance feedback, distills knowledge from top-ranked search results back into the model, strengthening the semantic mapping between vague descriptions and their intended targets.

The GradSelect framework: (a) Noise injection for denoising and (b) Gradient variance for term expansion.

Experimental Setup and Datasets

The performance of GradSelect was evaluated across three diverse benchmarks:

  • A-Cinderella: A clinical dataset from AphasiaBank featuring transcripts of patients retelling the Cinderella story. This is the most clinically relevant benchmark.
  • TREC-TOT 2023: A "Tip-of-the-Tongue" dataset focused on movie retrieval.
  • Reddit-TOMT: Real-world data from the "Tip of My Tongue" subreddit, serving as a proxy for natural word-retrieval difficulties.

Key Results

GradSelect demonstrated superior performance over standard retrieval models and large-scale language models:

  • Clinical Efficacy: On the A-Cinderella dataset, GradSelect achieved an Accuracy@5 of 64.96%, outperforming the base Dense Passage Retrieval (DPR) model (61.17%) and significantly surpassing GPT-4 (49.39%).
  • Retrieval Precision: In the TREC-TOT task, GradSelect improved the Mean Reciprocal Rank (MRR) from 0.2099 to 0.2456, indicating a higher frequency of correct targets appearing at the top of the results list.

Discussion and Limitations

A key insight from this research is that "off-the-shelf" giants like GPT-4 struggle with the nuanced, "tail-end" linguistic errors characteristic of clinical populations. Specialized, domain-specific robustification—like the surgical noise injection used in GradSelect—is essential for assistive technologies.

However, the method does have limitations. The reliance on gradient calculations increases computational overhead during training compared to simple prompting. Furthermore, the scarcity of large-scale clinical datasets remains a bottleneck. Future research will focus on optimizing inference speeds to enable real-time assistance for patients during active speech.

Summary updated

BibTeX
@inproceedings{kim2024intended,
  author = {Jongho Kim and Romain Storaï and Seung-won Hwang},
  title = {{Intended Target Identification for Anomia Patients with Gradient-based Selective Augmentation}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2024},
  doi = {10.18653/v1/2024.findings-emnlp.616},
}
Figure