Cloze Distillation: Improving Neural Language Models with Human Next-Word Prediction

Tiwalayo Eisape, Noga Zaslavsky, Roger Levy

Abstract

Modern neural language models (LMs) are highly effective at text generation but often exhibit a "dissociation" from human cognitive processing. This paper introduces Cloze Distillation (CD), a fine-tuning technique that leverages human "cloze" (fill-in-the-blank) data to align AI predictions with human expectations. By incorporating human behavioral data into the training objective, the authors demonstrate that LMs can better mimic human reading patterns, such as gaze duration, while maintaining or even improving standard language modeling performance.

Introduction: The Gap Between AI and Human Prediction

In psycholinguistics, Surprisal Theory posits that human reading time is proportional to the "surprisingness" of a word given its context. While state-of-the-art (SOTA) Large Language Models (LLMs) are trained to minimize perplexity (a measure of surprise) on massive corpora, they often fail to capture the "diffuse bets" humans make—the subconscious weighing of multiple semantic and syntactic possibilities. Interestingly, simpler models like LSTMs sometimes correlate better with human reading times than massive Transformers, despite having higher perplexity. This research addresses the challenge of imbuing neural models with these human-like predictive biases.

Methodology: Cloze Distillation

The authors propose Cloze Distillation (CD), a knowledge distillation framework where human behavioral data acts as the "teacher." The fine-tuning objective function ($L_i$) is a linear interpolation of two primary terms:

  1. KL Divergence ($D_i$): This measures the difference between the human cloze probability distribution and the model’s predicted distribution.
  2. Surprisal ($S_i$): The standard cross-entropy loss based on the actual next word in the corpus.

An interpolation coefficient ($\alpha$) balances these two terms, allowing for a controlled blend of human-like bias and standard corpus statistics.

Figure 1: The Cloze Distillation framework combines pre-trained LM predictions with human cloze data and ground truth corpus labels using an interpolation coefficient α.

Datasets and Evaluation

The primary dataset used is the Provo Corpus, which contains 55 paragraphs of text paired with:

  • Cloze Data: 40 human completions for each of the 2,689 word positions.
  • Eye-tracking Data: Gaze duration measurements from 84 native English speakers.

The baseline model is a 2-layer LSTM pre-trained on Wikitext-103. Evaluation is performed using a 55-fold cross-validation scheme across paragraphs to ensure generalization to held-out human data.

Key Insights and Results

The study reveals several critical findings regarding the alignment of LMs with human behavior:

  • The Perplexity Paradox: Lower perplexity on a corpus does not inherently lead to a better fit for human reading times. The initial LSTM baseline outperformed larger Transformers in raw correlation with gaze duration.
  • Optimal Interpolation: The best performance for both reading time prediction and standard language modeling was achieved at an intermediate $\alpha$ value (approximately 0.65). This indicates that human expectations and corpus statistics provide complementary signals.
  • Data Efficiency: Significant behavioral adaptation was achieved with only ~2,600 training samples, a tiny fraction of typical pre-training datasets.
Figure 2: Statistical comparison (F-statistic) of CD-LSTM against various baselines across different α values, highlighting the performance peak at intermediate levels.

Discussion and Limitations

Cloze-distilled LSTMs (CD-LSTM) showed statistically significant improvements in predicting human gaze duration and generalized well to held-out human cloze data. Surprisingly, fine-tuning on human predictions also slightly improved standard language modeling performance on the Provo corpus.

However, limitations remain. Collecting high-quality cloze data is resource-intensive, limiting the scale of this method. Furthermore, while CD-LSTMs predict reading times more accurately, it remains an open question whether they are capturing the underlying mechanisms of human language processing or merely approximating the behavioral outputs. Future work may involve applying Cloze Distillation during the pre-training phase of larger Transformer architectures.

Summary updated

Abstract

Contemporary autoregressive language models (LMs) trained purely on corpus data have been shown to capture numerous features of human incremental processing. However, past work has also suggested dissociations between corpus probabilities and human next-word predictions. Here we evaluate several state-of-the-art language models for their match to human next-word predictions and to reading time behavior from eye movements. We then propose a novel method for distilling the linguistic information implicit in human linguistic predictions into pre-trained LMs: Cloze Distillation. We apply this method to a baseline neural LM and show potential improvement in reading time prediction and generalization to held-out human cloze data.
BibTeX
@inproceedings{eisape2020cloze,
  author = {Tiwalayo Eisape and Noga Zaslavsky and Roger Levy},
  title = {{Cloze Distillation: Improving Neural Language Models with Human Next-Word Prediction}},
  booktitle = {CoNLL},
  year = {2020},
  doi = {10.18653/v1/2020.conll-1.49},
}
Figure