Abstract
Cloze tests are widely adopted in language exams to evaluate students' language proficiency. In this paper, we propose the first large-scale human-created cloze test dataset CLOTH, containing questions used in middle-school and high-school language exams. With missing blanks carefully created by teachers and candidate choices purposely designed to be nuanced, CLOTH requires a deeper language understanding and a wider attention span than previously automatically-generated cloze datasets. We test the performance of dedicatedly designed baseline models including a language model trained on the One Billion Word Corpus and show humans outperform them by a significant margin. We investigate the source of the performance gap, trace model deficiencies to some distinct properties of CLOTH, and identify the limited ability of comprehending the long-term context to be the key bottleneck.
Introduction
Reading comprehension is a cornerstone of human intelligence, and cloze tests—where participants fill in missing words in a passage—are a standard tool for evaluating this skill. While many automated datasets exist, they often rely on random blanking and simplistic distractors, allowing models to succeed using local syntactic patterns rather than deep understanding. We introduce CLOTH (CLOze test by TeacHers), a dataset designed to challenge models with the same rigor applied to human students by using questions crafted by professional educators.
The CLOTH Dataset
CLOTH consists of nearly 100,000 questions sourced from English exams for middle and high school students. Unlike previous benchmarks, these questions are curated by teachers to test specific linguistic and cognitive abilities. The dataset is split into CLOTH-M (middle school) and CLOTH-H (high school), with the latter presenting a higher level of difficulty in terms of vocabulary size and reasoning requirements.
Methodology
To evaluate current AI capabilities, we tested several baseline architectures, including LSTMs, Attentive Readers (Stanford AR and Position-aware AR), and a Language Model (LM) trained on the One Billion Word Corpus. We categorized questions into four primary reasoning types to better understand model performance:
- Grammar: Testing tense, prepositions, or voice.
- Short-term reasoning: Answerable using information within the same sentence.
- Matching/Paraphrasing: Finding a word used elsewhere in the context.
- Long-term reasoning: Requiring information synthesized from multiple sentences.
Additionally, we explored a Representative-based Model to bridge the gap between easy auto-generated data and hard teacher-designed data by identifying "human-like" questions for more effective pre-training.
Experimental Results
Our experiments reveal a substantial performance gap between state-of-the-art models and humans. While humans achieve an average accuracy of approximately 86%, the best-performing model (the 1B-LM utilizing three sentences of context) only reaches 70.7%. This gap is particularly pronounced in the high-school subset (CLOTH-H), where the complexity of distractors and the need for long-term context are greatest.
Key Insights
The "One-Sentence" Ceiling
Analysis shows that when humans are restricted to the sentence containing the blank, their performance drops to levels comparable to AI (around 71%). This suggests that the primary deficiency in current models is not vocabulary or syntax, but the inability to integrate information across multiple sentences effectively. While humans leverage the surrounding paragraph to reach 86% accuracy, models see only marginal gains from additional context.
Nuanced Distractors
Teachers design distractors to be "nuanced"—for example, ensuring all options are the same part of speech and grammatically plausible. This prevents models from using simple syntax checks to find the answer, requiring true semantic understanding to distinguish the correct choice from plausible alternatives.
Distributional Mismatch
Training a model purely on auto-generated data can actually hurt its performance on CLOTH. The patterns in "easy" auto-data are so different from "hard" teacher-data that knowledge transfer is limited, highlighting the necessity of high-quality, human-designed benchmarks for measuring true progress in NLP.
Conclusion
CLOTH serves as a challenging new benchmark that highlights the limitations of current NLP models in long-term reasoning and deep semantic understanding. Future work should focus on architectures capable of modeling long-range dependencies and incorporating external knowledge to bridge the significant gap between machine and human performance.
Summary updated