Dropout during inference as a model for neurological degeneration in an image captioning network

Bai Li, Ran Zhang, Frank RudziczarXiv 2018 · arXiv

Abstract

This study explores the intersection of deep learning and clinical neurology by simulating neurodegenerative language impairment through intentional model degradation. By applying dropout during the inference phase of a healthy image captioning network, the authors provide a computational window into how diseases like Alzheimer’s disrupt human language. The resulting linguistic patterns—including repetitive words and nonsensical "jargon speech"—qualitatively mirror the speech of patients with Alzheimer’s disease and Wernicke’s aphasia.

Introduction

In clinical neurology, describing a picture (such as the "Cookie Theft" task) is a standard diagnostic tool. Patients with Alzheimer’s Disease (AD) or Wernicke’s Aphasia (WA) often struggle with this task; their speech may become repetitive, lose semantic meaning, or devolve into "jargon"—fluent-sounding but meaningless strings of words. While many models generate "perfect" language, fewer computational models explain how language degrades. This research addresses this gap by testing whether the stochastic nature of dropout—a common regularization technique—can serve as a proxy for the inhibited electrochemical signal transmission and neuron loss seen in neurodegenerative disorders.

Methodology

The authors implemented a standard "Show and Tell" image captioning architecture comprising a pre-trained VGG16 Convolutional Neural Network (CNN) as an encoder and a Gated Recurrent Unit (GRU) Recurrent Neural Network (RNN) as a decoder.

Normally, dropout is used only during training to prevent overfitting. In this study, the authors introduced a constant level of dropout ($d_e$) to the hidden state of the GRU during the inference phase. By randomly setting a percentage of neuron outputs to zero at each time step, they simulated the "noisy" or failing signal transmission characteristic of a degenerating brain. The model was trained and evaluated on the COCO2014 dataset, with evaluation focusing on how output changed as the inference dropout rate increased from 0.0 to 0.8.

Results and Discussion

The study quantified the collapse of language across several metrics, as shown in the table below.

Quantitative performance and linguistic metrics across varying inference dropout rates ($d_e$).

Performance Decay and Rambling

As the inference dropout ($d_e$) increased, performance metrics plummeted. For a model trained without dropout ($d_t=0.0$), the BLEU-4 score dropped from 20.1 to 0.2 as $d_e$ reached 0.8. Furthermore, at high dropout levels ($d_e \geq 0.6$), the model began to "ramble," with the probability of sentences exceeding 20 words ($p(len > 20)$) approaching 100%, indicating a failure to reach a logical conclusion.

The Diversity Paradox

Interestingly, as dropout increases, the model’s vocabulary size ($|V|$) increases dramatically (e.g., from 733 to 9585). This does not represent healthy diversity; rather, it indicates the model losing its "train of thought" and sampling random, low-probability words, mimicking the "jargon speech" seen in aphasia. The authors noted a "sweet spot" at $d_e = 0.4$, where the Kullback–Leibler divergence ($D_{KL}$) was minimized, suggesting that a moderate amount of noise forces the model to use a broader range of words from the training corpus distribution.

Qualitative Error Patterns

The authors identified two primary error patterns matching clinical observations:

  1. Perseveration: Repeating the same word (e.g., "...yellow chihuahua chihuahua chihuahua").
  2. Paraphasia/Jargon: Transitioning into nonsense (e.g., "...won while too frown").

Conclusion

While dropout is a relatively blunt instrument compared to the complexities of biological neurodegeneration, it serves as a powerful functional proxy. The most promising application of this technique is in data augmentation. By generating "synthetic" impaired speech, researchers can train diagnostic AI to detect early-stage dementia in real patients more effectively, even when clinical data is scarce. Future work may explore more nuanced degradation methods, such as adding Gaussian noise or targeted neuron ablation.

Summary updated

Abstract

We replicate a variation of the image captioning architecture by Vinyals et al. (2015), then introduce dropout during inference mode to simulate the effects of neurodegenerative diseases like Alzheimer's disease (AD) and Wernicke's aphasia (WA). We evaluate the effects of dropout on language production by measuring the KL-divergence of word frequency distributions and other linguistic metrics as dropout is added. We find that the generated sentences most closely approximate the word frequency distribution of the training corpus when using a moderate dropout of 0.4 during inference.
BibTeX
@misc{li2018dropout,
  author = {Bai Li and Ran Zhang and Frank Rudzicz},
  title = {{Dropout during inference as a model for neurological degeneration in an image captioning network}},
  year = {2018},
  eprint = {1808.03747},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1808.03747},
}
Figure