Perhaps PTLMs Should Go to School - A Task to Assess Open Book and Closed Book QA

Manuel R. Ciosici, Joe Cecil, Alex Hedges, Dong-Ho Lee, Marjorie Freedman, Ralph M. Weischedel

Introduction

Researchers have introduced LEFT (Learning from Textbooks), a new benchmark designed to test whether pre-trained language models (PTLMs) can actually "learn" from academic materials. The study reveals a sobering reality: even after "reading" a college-level textbook through continued pre-training, state-of-the-art models like T5 and GPT-Neo barely perform better than random guessing on closed-book exams.

The Challenge of Domain-Specific Knowledge

Current AI models are incredibly good at answering questions based on general knowledge found across the internet. However, they struggle with domain-specific, "long-tail" information that isn't repeated millions of times in their training data.

Most existing Question Answering (QA) benchmarks fall into two camps: they either provide the answer in a short snippet (reading comprehension) or rely on the model's massive "memory" of the web. There is a gap in evaluating how well a model can ingest a new, structured source of information—like a textbook—and then apply that knowledge. For AI to be truly useful in specialized fields like law or medicine, we need to move beyond general web-crawled knowledge and toward intentional learning from authoritative documents.

The LEFT Benchmark

The benchmark, LEFT, utilizes two openly licensed college-level textbooks from OpenStax: American Government 2e (AG) and U.S. History (USH).

Statistics of the LEFT benchmark datasets, including chapter counts and statement totals.

The test set consists of hundreds of True/False statements manually derived from the textbooks' own review questions. Crucially, the authors rewrote these questions into pairs of true and false statements with high word overlap. This prevents the models from "cheating" by simply looking for familiar keywords rather than understanding the underlying facts.

Evaluation Framework

The authors propose three distinct evaluation configurations to mimic how a human student might interact with a course:

  1. Prior-knowledge: The model takes the test "out of the box" (fine-tuned only on general QA tasks like BoolQ). This measures what the model already knows from its initial pre-training on the web.
  2. Closed-book (after reading): The model is given the textbook to "read" via continued pre-training (masking and predicting words in the textbook). It then takes the test without access to the text, relying on what it "learned."
  3. Open-book: The model can access the textbook during the test. The researchers tested both automated retrieval (using sBERT) and "Gold IR," where the model is handed the exact paragraph containing the answer.

The researchers used T5-3B and GPT-Neo 2.7B as their baseline models, fine-tuning them on the BoolQ dataset to handle the True/False format of the LEFT task.

Experimental Results

The most striking insight is the "Memorization vs. Understanding" gap: while models can be trained to "memorize" the textbook (improving their ability to predict the next word in a sentence), this doesn't translate to answering questions.

Performance of T5 and GPT-Neo models across Prior-knowledge, Closed-book, and Open-book settings.

Key Findings:

  • Prior Knowledge: Both T5 and GPT-Neo performed at approximately 50% accuracy—essentially the same as flipping a coin. This confirms the textbook content was not well-represented in their original training data.
  • Closed-book (After Reading): T5 showed a minor improvement to 56%, while GPT-Neo remained near 50-55%. This indicates that "reading" the book via standard language modeling objectives is an inefficient way for AI to learn facts.
  • Open-book (Gold IR): When provided with the correct paragraph, T5 reached its peak performance of ~71-74%. While much better, it is still far from "solving" the task, suggesting that even with the answer in front of them, these models struggle with the nuances of college-level social sciences.

Retrieval is clearly the current ceiling. Performance jumps significantly in the "Open-book" setting, especially with "Gold IR." This suggests that the bottleneck isn't just the model's reasoning ability, but its inability to internalize and recall specific facts without a "cheat sheet."

Discussion and Future Outlook

The authors acknowledge several constraints, such as a limited 128-token context window due to memory limitations, which may have cut off relevant information during open-book tasks.

Future work will involve expanding the dataset with more instructor-written questions and categorizing the types of knowledge required (e.g., temporal reasoning vs. definition lookup). The results suggest that "learning" from text might require more sophisticated algorithms than simple next-word prediction.

Summary updated

BibTeX
@inproceedings{ciosici2021perhaps,
  author = {Manuel R. Ciosici and Joe Cecil and Alex Hedges and Dong-Ho Lee and Marjorie Freedman and Ralph M. Weischedel},
  title = {{Perhaps PTLMs Should Go to School - A Task to Assess Open Book and Closed Book QA}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2021},
  doi = {10.18653/v1/2021.emnlp-main.493},
}
Figure