Intended Target Identification for Anomia Patients with Gradient-based Selective Augmentation

Jongho Kim, Romain Storaï, Seung-won Hwang

Introduction

Anomia is a prevalent symptom of aphasia characterized by the inability to retrieve specific words despite knowing the intended concept. To compensate, individuals often employ circumlocution, or "talking around" the target item. While helpful for human listeners, circumlocution poses significant challenges for automated language models (LMs). Specifically, these models face two primary hurdles: Word Retrieval Failure (C1), where essential keywords are entirely absent, and Word Retrieval Error (C2), where patients substitute the target with semantically related but incorrect terms, known as semantically paraphasic errors (SPE).

Challenges in identifying target items from circumlocution, illustrating missing keywords (C1) and substituted errors (C2).

The GradSelect Framework

To address these challenges, researchers introduced GradSelect, a novel data augmentation framework that leverages gradient signals from the model to guide selective denoising and vocabulary expansion. Unlike traditional augmentation methods that apply random perturbations, GradSelect uses the model's internal feedback to distinguish between critical semantic signals and distracting errors.

Selective Denoising via Circumlocution Augmentation

GradSelect calculates the gradient magnitude for each token in a circumlocution. The researchers observed that tokens with the highest gradients often represent "unperturbed keywords"—the few correct clues provided by the patient. To improve model robustness against SPE (C2), GradSelect injects noise into other high-impact terms while preserving these critical keywords. This "surgical" approach teaches the model to ignore common speech errors without losing the core semantic signal.

Vocabulary Expansion via Item Augmentation

To combat Word Retrieval Failure (C1), GradSelect identifies top-ranked candidate items that may contain the missing relevant terms. It then employs Self-Knowledge Distillation (Self-KD). In this process, a teacher model's predictions on these expanded items guide the training of a student model, effectively bridging the gap between a patient's vague description and the correct, unseen concepts.

The GradSelect process: (a) selective noise injection based on gradient values and (b) item expansion based on gradient variance.

Experimental Evaluation

The framework was evaluated using several benchmarks, including the Reddit-TOMT and TREC-TOT "Tip-of-the-Tongue" datasets, which simulate retrieval difficulties. Crucially, it was also tested on A-cinderella, a clinical dataset from AphasiaBank containing transcripts of real patients. A "Challenge Set" was further developed by removing repetitive "retracing" words to prevent the model from exploiting simple linguistic leaks.

Key Results and Insights

GradSelect demonstrated superior performance across all metrics. On the TREC-TOT dataset, it improved the nDCG@10 score from a baseline of 0.2341 to 0.2695. In clinical scenarios, GradSelect achieved an accuracy at 5 (acc@5) of 54.20% on the A-cinderella challenge set, notably outperforming GPT-4, which reached only 43.95%.

A key insight from the study is that gradient values serve as an effective proxy for token importance. Removing tokens with the highest gradients led to a sharp decline in performance, confirming that the model relies heavily on these "unperturbed keywords." Furthermore, the specialized, smaller retriever fine-tuned with GradSelect proved more resilient to the unique linguistic patterns of aphasic speech than much larger, general-purpose models.

Conclusion

GradSelect offers a robust solution for identifying intended targets in the presence of speech errors and missing information. By utilizing gradient-based selective augmentation, the framework effectively mitigates the impact of semantically paraphasic errors and expands the model's reach to unseen terms. Future work will explore inference-time denoising and advanced prompting strategies to further enhance clinical utility.

Summary updated

BibTeX
@inproceedings{kim2024intended,
  author = {Jongho Kim and Romain Storaï and Seung-won Hwang},
  title = {{Intended Target Identification for Anomia Patients with Gradient-based Selective Augmentation}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2024},
  doi = {10.18653/v1/2024.findings-emnlp.616},
}
Figure