TL;DR
The Learning from Textbooks (LEFT) benchmark challenges AI models to "go to school" by reading college-level textbooks and answering true/false questions. The results are a wake-up call: even state-of-the-art models like T5 and GPT-Neo struggle to move beyond random guessing after "reading" the material, suggesting that current pre-training methods fail to capture deep, domain-specific understanding.
What Problem Does This Paper Address?
While Pre-trained Language Models (PTLMs) like GPT-3 and T5 are incredibly good at answering general knowledge questions, they rely on facts being repeated millions of times across the web. However, specialized, domain-specific knowledge—the kind found in a technical manual or a college textbook—is rarely repeated often enough to be "memorized" during standard pre-training.
The authors identify a major gap: we lack a way to measure how well a model can learn from a single, authoritative, but relatively small source of information. They propose the LEFT task to evaluate whether models can actually "learn by reading" rather than just relying on the statistical patterns of their massive initial training sets.
Methods
The authors introduced three distinct evaluation configurations to simulate different learning stages:
- Prior-knowledge: The model takes the test "cold," relying only on its original pre-training. This measures what the model already knows.
- Closed-book (after reading): The model is allowed to "read" the textbook (via continued light pre-training on the text) but cannot look at the book while answering questions. This tests the model's ability to internalize and reason over new information.
- Open-book: The model can access specific textbook paragraphs while answering. This tests reading comprehension and information retrieval.
To establish baselines, they used T5-3B and GPT-Neo 2.7B, fine-tuning them on the BoolQ dataset (a standard yes/no question-answering task) to teach the models the format of the task without giving them the content of the textbooks.
Datasets
The LEFT benchmark utilizes two openly licensed, college-level introductory textbooks:
- American Government 2e (AG)
- U.S. History (USH)
The researchers manually rewrote multiple-choice review questions from these books into balanced pairs of true/false statements. These pairs were designed with high word overlap to prevent models from "cheating" by simply looking for familiar keywords.
The dataset statistics are summarized in the table below:
Key Insights and Ideas
- PTLMs are "Tabula Rasa" for specialized domains: Out-of-the-box models performed at roughly 50% accuracy—no better than a coin flip. This confirms that the specific facts in these textbooks are not effectively captured by general web-scale pre-training.
- Reading $\neq$ Understanding: Simply performing more pre-training on the textbook (the "Closed-book" setting) yielded only marginal improvements (around 56% at best). The authors suggest that while models might learn to complete sentences from the book, they aren't "understanding" the subject matter well enough to classify novel statements.
- The Retrieval Gap: In the Open-book setting, there is a significant performance gap between using automated retrieval (sBERT) and "Gold" retrieval (manually selected relevant paragraphs). This indicates that finding the right information is still a major bottleneck for AI "students."
- The ~70% Ceiling: Even when given the exact paragraph containing the answer (Open-book + goldIR), the best model only reached ~74% accuracy. This suggests that the challenge isn't just finding the data, but reasoning about the relationship between the question and the text.
Key Results
The baseline results highlight the difficulty of the task across all settings. As shown in Figure 2, the jump from Prior-knowledge to Open-book is significant, but the Closed-book "learning" phase remains stubbornly difficult.
- Prior-knowledge: Both T5 and GPT-Neo hovered around 50%.
- Closed-book: T5 improved slightly to 56.45% on the AG Dev set after "reading" the book.
- Open-book (GoldIR): T5 achieved its peak performance of 74.30% on the AG Test set when provided with the correct context.
Limitations and Future Work
The authors acknowledge several hurdles:
- Technical Constraints: Due to memory limits, the models were restricted to a maximum sequence length of 128 word pieces, which may have truncated necessary context.
- "Reading" Algorithms: The current method of "reading" (continued pre-training) is quite basic. Future work could explore more sophisticated ways to encode knowledge, such as extracting and storing triples (knowledge graphs) during the reading phase.
- Knowledge Categorization: The team plans to categorize the types of knowledge required for each question (e.g., temporal reasoning vs. factual recall) to pinpoint exactly where language models fail.
The LEFT leaderboard is available for the community to track progress on this challenging new "school" for AI.
Summary updated