Abstract: Cloze tests are widely adopted in language exams to evaluate students' language proficiency. In this paper, we propose the first large-scale human-created cloze test dataset CLOTH, containing questions used in middle-school and high-school language exams. With missing blanks carefully created by teachers and candidate choices purposely designed to be nuanced, CLOTH requires a deeper language understanding and a wider attention span than previously automatically-generated cloze datasets. We test the performance of dedicatedly designed baseline models including a language model trained on the One Billion Word Corpus and show humans outperform them by a significant margin. We investigate the source of the performance gap, trace model deficiencies to some distinct properties of CLOTH, and identify the limited ability of comprehending the long-term context to be the key bottleneck.
Introduction
In the field of Natural Language Processing (NLP), "cloze tests" (fill-in-the-blank exercises) are a standard benchmark for measuring reading comprehension. However, most existing large-scale datasets are generated automatically—for example, by randomly deleting named entities from news summaries. The authors argue this approach leads to three major flaws: triviality (blanks are too easy), ambiguity (multiple words fit equally well), and a lack of linguistic purpose.
To address these issues, the researchers introduce CLOTH (CLOze test by TeacHers), a dataset sourced from real-world English exams designed by educators to target specific linguistic phenomena like grammar, logic, and vocabulary nuances.
The CLOTH Dataset
CLOTH consists of 99,433 questions across 7,131 passages. It is categorized into two subsets: CLOTH-M (middle school) and CLOTH-H (high school), reflecting the increasing complexity of language education. Unlike machine-generated datasets, CLOTH includes "distractors" (wrong options) that are often grammatically correct and semantically related to the right answer, requiring the reader to understand the broader narrative context to choose correctly.
Experimental Methods
The authors evaluated several architectures to establish a performance baseline:
- Baselines: Standard LSTMs and Attentive Readers (AR) which use attention mechanisms to locate relevant context.
- Language Models (LM): A model trained on the "One Billion Word Corpus" (1B-LM), tested with both single-sentence and three-sentence context windows.
- Representativeness-based Model: A semi-supervised approach where a discriminator identifies which words in raw text are likely to be chosen as blanks by a teacher. This model weights training data based on these "teacher-like" qualities, achieving 58.3% accuracy compared to 48.4% for a basic LSTM.
Key Results and Analysis
The results highlight a substantial performance gap between state-of-the-art models and human readers. Humans achieved an accuracy of approximately 86%, whereas the best-performing model (1B-LM with a three-sentence context) reached only 70.7%.
The Long-term Reasoning Bottleneck
A critical insight from the study is the "long-term bottleneck." The authors categorized questions into types such as Grammar, Short-term reasoning, and Long-term reasoning. While models are increasingly proficient at local context (within a single sentence), they struggle significantly when the answer requires synthesizing information from several sentences away. Even when the context window for the 1B-LM was increased, performance gains were marginal, suggesting that current models lack the ability to perform deep logical reasoning over extended passages.
Conclusion
CLOTH serves as a more "honest" measure of reading comprehension than automated datasets. By utilizing teacher-created content, it exposes the limitations of current AI in handling nuanced vocabulary and long-term reasoning. Future work will need to focus on designing attention models oriented toward reasoning rather than simple pattern matching and incorporating external knowledge to better handle sophisticated distractors.
Summary updated