Abstract: We explore a novel approach to reading compliance, leveraging large language models to select inline challenges that discourage skipping during reading. This lightweight 'testing' is accomplished through automatically identified context clozes where the reader must supply a missing word that would be hard to guess if earlier material was skipped. Clozes are selected by scoring each word by the contrast between its likelihood with and without prior sentences as context, preferring to leave gaps where this contrast is high. We report results of an initial human-participant test that indicates this method can find clozes that have this property.
Introduction
A persistent challenge in higher education is ensuring students complete assigned readings. When students skip readings, instructors are often forced to spend valuable class time reteaching textbook material rather than engaging in higher-level discussion. While traditional quizzes can enforce compliance, they are labor-intensive to create and prone to leaking online. Alternative methods, such as social reading platforms, require significant student output and peer interaction, which may not always be desirable.
This paper proposes a lightweight, automated alternative: the "context cloze." By embedding fill-in-the-blank questions directly into the reading material, we can verify compliance in real-time. The technical challenge lies in selecting the correct words to blank out—specifically, words that are predictable only if the preceding context has been read.
The Context Cloze Method
The core of our approach is using Large Language Models (LLMs) as proxies for human readers. We utilize GPT-2 to score potential target words based on their predictability in two scenarios: with and without the preceding sentence.
For a target word, we calculate its probability given only the current partial sentence ($t_0$) and its probability given the prior sentence plus the partial sentence ($t_1$). To ensure the cloze is effective, we also identify a "confuser" word—a distractor that is highly probable without context ($c_0$) but becomes improbable once the prior context is introduced ($c_1$).
We use a composite scoring formula to identify optimal cloze locations: $$s = (c_0 - t_0) + (t_1 - c_1) + (t_1 - t_0)$$
This formula prioritizes instances where the target word's likelihood increases significantly with context while the distractor's likelihood decreases. To better simulate a student's baseline knowledge, we prepended a Wikipedia summary of the subject matter to the LLM's prompt, grounding its predictions in the relevant domain.
Experimental Evaluation
To validate our method, we conducted a human-participant study with 100 users via Prolific. We tested two datasets:
- CPRAG20: A baseline set of 20 sentences from a classic psycholinguistics study.
- PSYCH50: A set of 50 high-scoring clozes automatically extracted from an open-source Psychology textbook using our GPT-2 scoring method.
Participants were asked to predict the missing word both with and without the preceding sentence as context. This allowed us to measure the "predictability gain" provided by the context.
Results
The results confirm that LLM-selected clozes effectively mirror human reading behavior. On the baseline CPRAG20 dataset, human predictability rose from 5.7% to 71.9% with context. More importantly, on the automatically generated PSYCH50 dataset, predictability increased from 19.3% to 47.4%—a mean absolute increase of 28.1%.
As shown in Figure 1, the vast majority of the 50 selected sentences showed the intended gain. While a small number of prompts showed a decrease in predictability, the overall trend demonstrates that the automated selection method successfully identifies words that require contextual knowledge to solve.
Discussion and Future Work
Our findings suggest that LLMs can effectively automate the generation of reading compliance checks. This method is lightweight, integrates seamlessly into the reading experience, and avoids the pitfalls of manually created quizzes.
Future research will focus on expanding the context window beyond a single preceding sentence and fine-tuning models on specific textbooks to improve domain-aware scoring. Additionally, we plan to deploy this system in a mobile reading application to evaluate its impact on student learning outcomes and reading habits in real-world classroom settings.
Summary updated