Perhaps PTLMs should go to School – A Task to Assess Open Book and Closed Book QA

Manuel Rafael Ciosici, Joe Cecil, Dong-Ho Lee, Alex Hedges, Marjorie Freedman, Ralph Weischedel

TL;DR

The Learning from Textbooks (LEFT) benchmark challenges AI models to "go to school" by reading college-level textbooks and answering true/false questions. The results are a wake-up call: even state-of-the-art models like T5 and GPT-Neo struggle to move beyond random guessing after "reading" the material, suggesting that current pre-training methods fail to capture deep, domain-specific understanding.

What Problem Does This Paper Address?

While Pre-trained Language Models (PTLMs) like GPT-3 and T5 are incredibly good at answering general knowledge questions, they rely on facts being repeated millions of times across the web. However, specialized, domain-specific knowledge—the kind found in a technical manual or a college textbook—is rarely repeated often enough to be "memorized" during standard pre-training.

The authors identify a major gap: we lack a way to measure how well a model can learn from a single, authoritative, but relatively small source of information. They propose the LEFT task to evaluate whether models can actually "learn by reading" rather than just relying on the statistical patterns of their massive initial training sets.

Methods

The authors introduced three distinct evaluation configurations to simulate different learning stages:

  1. Prior-knowledge: The model takes the test "cold," relying only on its original pre-training. This measures what the model already knows.
  2. Closed-book (after reading): The model is allowed to "read" the textbook (via continued light pre-training on the text) but cannot look at the book while answering questions. This tests the model's ability to internalize and reason over new information.
  3. Open-book: The model can access specific textbook paragraphs while answering. This tests reading comprehension and information retrieval.

To establish baselines, they used T5-3B and GPT-Neo 2.7B, fine-tuning them on the BoolQ dataset (a standard yes/no question-answering task) to teach the models the format of the task without giving them the content of the textbooks.

Datasets

The LEFT benchmark utilizes two openly licensed, college-level introductory textbooks:

  • American Government 2e (AG)
  • U.S. History (USH)

The researchers manually rewrote multiple-choice review questions from these books into balanced pairs of true/false statements. These pairs were designed with high word overlap to prevent models from "cheating" by simply looking for familiar keywords.

The dataset statistics are summarized in the table below:

Data overview for the AG and USH textbooks

Key Insights and Ideas

  • PTLMs are "Tabula Rasa" for specialized domains: Out-of-the-box models performed at roughly 50% accuracy—no better than a coin flip. This confirms that the specific facts in these textbooks are not effectively captured by general web-scale pre-training.
  • Reading $\neq$ Understanding: Simply performing more pre-training on the textbook (the "Closed-book" setting) yielded only marginal improvements (around 56% at best). The authors suggest that while models might learn to complete sentences from the book, they aren't "understanding" the subject matter well enough to classify novel statements.
  • The Retrieval Gap: In the Open-book setting, there is a significant performance gap between using automated retrieval (sBERT) and "Gold" retrieval (manually selected relevant paragraphs). This indicates that finding the right information is still a major bottleneck for AI "students."
  • The ~70% Ceiling: Even when given the exact paragraph containing the answer (Open-book + goldIR), the best model only reached ~74% accuracy. This suggests that the challenge isn't just finding the data, but reasoning about the relationship between the question and the text.

Key Results

The baseline results highlight the difficulty of the task across all settings. As shown in Figure 2, the jump from Prior-knowledge to Open-book is significant, but the Closed-book "learning" phase remains stubbornly difficult.

Baseline accuracy across evaluation settings
  • Prior-knowledge: Both T5 and GPT-Neo hovered around 50%.
  • Closed-book: T5 improved slightly to 56.45% on the AG Dev set after "reading" the book.
  • Open-book (GoldIR): T5 achieved its peak performance of 74.30% on the AG Test set when provided with the correct context.

Limitations and Future Work

The authors acknowledge several hurdles:

  • Technical Constraints: Due to memory limits, the models were restricted to a maximum sequence length of 128 word pieces, which may have truncated necessary context.
  • "Reading" Algorithms: The current method of "reading" (continued pre-training) is quite basic. Future work could explore more sophisticated ways to encode knowledge, such as extracting and storing triples (knowledge graphs) during the reading phase.
  • Knowledge Categorization: The team plans to categorize the types of knowledge required for each question (e.g., temporal reasoning vs. factual recall) to pinpoint exactly where language models fail.

The LEFT leaderboard is available for the community to track progress on this challenging new "school" for AI.

Summary updated

Abstract

Our goal is to deliver a new task and leaderboard to stimulate research on question answering and pre-trained language models (PTLMs) to understand a significant instructional document, e.g., an introductory college textbook or a manual. PTLMs have shown great success in many question-answering tasks, given significant supervised training, but much less so in zero-shot settings. We propose a new task that includes two college-level introductory texts in the social sciences (American Government 2e) and humanities (U.S. History), hundreds of true/false statements based on review questions written by the textbook authors, validation/development tests based on the first eight chapters of the textbooks, blind tests based on the remaining textbook chapters, and baseline results given state-of-the-art PTLMs. Since the questions are balanced, random performance should be ~50%. T5, fine-tuned with BoolQ achieves the same performance, suggesting that the textbook’s content is not pre-represented in the PTLM. Taking the exam closed book, but having read the textbook (i.e., adding the textbook to T5’s pre-training), yields at best minor improvement (56%), suggesting that the PTLM may not have “understood” the textbook (or perhaps misunderstood the questions). Performance is better (~60%) when the exam is taken open-book (i.e., allowing the machine to automatically retrieve a paragraph and use it to answer the question).
BibTeX
@inproceedings{ciosici2021perhaps,
  author = {Manuel Rafael Ciosici and Joe Cecil and Dong-Ho Lee and Alex Hedges and Marjorie Freedman and Ralph Weischedel},
  title = {{Perhaps PTLMs should go to School – A Task to Assess Open Book and Closed Book QA}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2021},
  doi = {10.18653/v1/2021.emnlp-main.493},
}
Figure