Introduction
As Large Language Models (LLMs) like GPT-4 are increasingly adopted for "cultural analytics"—the computational study of literature and narrative—a critical methodological flaw has emerged: the lack of transparency regarding training data. If a model has already encountered a literary work during its training phase, using that same work to evaluate its analytical or reasoning capabilities constitutes data contamination. This study performs a "data archaeology" to uncover the extent of book memorization within OpenAI's black-box models.
Methodology: The Name Cloze Task
To detect membership inference, the researchers developed a name cloze task. They extracted short passages (40–60 tokens) from 571 works of fiction and masked a single proper name. These passages were specifically selected because the immediate context does not provide sufficient clues to identify the character. Consequently, a human or a model without prior exposure to the text would have a near-zero success rate. Correct identification of the name (e.g., "Okonkwo" in a passage from Things Fall Apart) serves as a high-confidence signal of memorization.
Data Sources
The evaluation spanned 571 English-language novels published between 1749 and 2020, categorized into several distinct groups:
- LitBank: Public domain novels (pre-1923).
- Pulitzer Nominees: Award-recognized works (1924–2020).
- Bestsellers: High-circulation titles from NYT and Publishers Weekly.
- Diverse Voices: Novels from the Black Book Interactive Project (BBIP) and BCALA, alongside Global Anglophone works.
- Genre Fiction: Representative works from Sci-Fi/Fantasy, Horror, Mystery, and Romance.
Key Findings: The Depth of Memorization
The study reveals that memorization is not uniform but is instead a function of a book's "web presence." Titles frequently discussed on fan wikis, Goodreads, or educational sites like LitCharts are deeply ingrained in the models' weights.
GPT-4 consistently demonstrates a higher degree of memorization than ChatGPT. For public domain works, GPT-4 achieved a name cloze accuracy of 24.4%, more than triple ChatGPT’s 7.2%. Notably, Science Fiction and Fantasy works are disproportionately represented among the most memorized copyrighted texts, accounting for 12 of the top 20 titles.
Impact on Downstream Tasks
The most significant finding is the "cheating" effect on analytical tasks. When asked to predict a book's publication year, GPT-4's accuracy was tied directly to its level of memorization.
For the top 10% most memorized books, GPT-4 achieved a Mean Absolute Error (MAE) of just 0.6 years. In contrast, for the bottom 10% (least memorized), the error surged to 14.5 years. This discrepancy proves that the model is often performing fact retrieval rather than linguistic or historical inference.
Conclusion
The "data contamination" identified here poses a validity crisis for using LLMs in literary research. When models "bleed" memorized data into new contexts, they introduce significant biases. The authors argue that the path forward for cultural analytics requires a shift toward open-source models with fully transparent, auditable training datasets to ensure the scientific integrity of computational literary studies.
Summary updated