Quizbowl: The Case for Incremental Question Answering

Pedro Rodriguez, Shi Feng, Mohit Iyyer, He He, Jordan L. Boyd-Graber

TL;DR

This paper introduces Quizbowl as a rigorous benchmark for incremental question answering (QA), where models must process text word-by-word and decide exactly when to interrupt and answer. By curating the massive QANTA dataset—containing over 100,000 academic trivia questions and millions of human gameplay logs—the authors develop a decoupled "guesser-buzzer" framework that successfully balances QA accuracy with sequential decision-making under uncertainty, even competing live against world-class human trivia players.

What Problem Does This Paper Address?

Traditional question answering benchmarks evaluate models on static text or full questions. Even famous benchmarks like Jeopardy! (which IBM Watson famously mastered) evaluate models only after a clue is fully read. This format turns the competition into a test of electronic reflex speed rather than a true reflection of incremental comprehension.

In contrast, the scholastic game of Quizbowl relies on pyramidal questions: sentences are revealed word-by-word, starting with highly obscure clues and ending with obvious "giveaways."

The historical evolution of trivia formats toward more professionalized, fair structures like Quizbowl.

This format introduces a complex sequential decision-making problem. A player must continuously weigh the benefit of buzzing early (to beat the opponent) against the risk of buzzing incorrectly (which incurs a point penalty and locks them out of the question). To succeed, a machine learning system must solve three distinct challenges simultaneously:

  1. Factoid QA over a massive open-domain output space (thousands of Wikipedia entities).
  2. Model Calibration to accurately estimate its own confidence dynamically.
  3. Strategic Buzzing that factors in the state of the game and opponent behavior.

Methods

The authors address these challenges by splitting the system into two distinct, semi-independent components:

  1. The Guesser: This component handles the core QA task. Given the text revealed so far, it outputs a probability distribution over roughly 25,000 candidate Wikipedia answers. The authors experiment with several architectures, including traditional Information Retrieval (BM25), linear models, Deep Averaging Networks (DANs), Recurrent Neural Networks (GRUs), and BERT.
  2. The Buzzer: Because neural network softmax outputs are notoriously poorly calibrated, the authors treat the buzzer as a separate meta-classifier. Operating as a binary classifier (buzz vs. wait) at every word, the buzzer processes the dynamics of the guesser's outputs over time. It extracts features such as the raw scores of the top guesses, the margin between the first and second guess, and how much the top rankings fluctuate from word to word. This state history is fed into an MLP or an LSTM to make the final decision.

Datasets

The authors introduce the QANTA Dataset, which contains two primary components:

  • Questions: 120,000 human-authored, syntactically complex Quizbowl questions. Because each question contains multiple pyramidal clues, they can be broken down into over 650,000 sentence-level QA pairs spanning 11.4 million tokens.
  • Gameplay Records: A unique dataset of 3.9 million filtered logs from humans playing Quizbowl online. This records exactly which word triggered a human player's buzz, whether they were right or wrong, and their overall skill level, providing an empirical baseline for training the buzzer to out-speed humans.

Key Insights and Ideas

  • Trivia as an Adversarial Arms Race: Unlike crowdsourced datasets (e.g., SQuAD), Quizbowl questions are written by domain experts explicitly designed to defeat superficial pattern matching. This makes the dataset naturally robust against the cheap linguistic shortcuts that language models frequently exploit.
  • Decoupling for Better Calibration: Instead of relying on a model's raw confidence score, tracking the trajectory of guesses over time (e.g., watching a model vacillate between answers before stabilizing) provides a much stronger signal for whether the model is actually correct.
  • The Perils of Order-Awareness: While transformer models like BERT excel at understanding early, obscure clues due to their contextual awareness, they can lag behind simpler bag-of-words models near the end of a question because they over-emphasize early sentence structures during full-question inference.

Key Results

  • Guesser Accuracy: At the start of a question (after sentence one), BERT achieved the highest accuracy at 9.49%, outperforming the DAN (7.88%) and BM25 (6.23%). However, by the end of the question, the traditional Information Retrieval (BM25) baseline actually won out with 54.5% accuracy, compared to BERT's 48.8%.
  • Buzzer Performance: The RNN-based buzzer significantly outperformed a simple static threshold strategy. On a metric measuring Expected Wins (EW) against an average human player, the RNN buzzer scored 0.302 (where a theoretically perfect oracle buzzer scores 0.502), proving highly effective at mitigating costly, over-aggressive premature buzzes.
  • Live Matches: The combined system was put to the test in live, public exhibition matches against elite human trivia players, successfully securing narrow victories in events like the 2017 High School National Championship Tournament (HSNCT).

Limitations and Future Work

  • Data Sparsity: The distribution of trivia topics has a massive long tail. Nearly 18% of the test questions featured answers that never appeared in the training set, highlighting a severe need for better few-shot learning and domain adaptation techniques.
  • Fragility to Phrasing: The models remain highly sensitive to exact wording. The authors demonstrated that simple synonym substitution attacks (swapping words like "phenomenon" for "event") easily trick the models into changing their guesses.
  • Lack of Explicit Opponent Modeling: The current buzzer assumes a "locally optimal" strategy based purely on its own correctness. It does not dynamically adjust its aggressiveness based on the specific opponent's historical speed or whether its team is currently winning or losing.

Summary updated

Abstract

Scholastic trivia competitions test knowledge and intelligence through mastery of question answering. Modern question answering benchmarks are one variant of the Turing test. Specifically, answering a set of questions as well as a human is a minimum bar towards demonstrating human-like intelligence. This paper makes the case that the format of one competition -- where participants can answer in the middle of hearing a question (incremental) -- better differentiates the skill between (human or machine) players. Additionally, merging a sequential decision-making sub-task with question answering (QA) provides a good setting for research in model calibration and opponent modeling. Thus, embedded in this task are three machine learning challenges: (1) factoid QA over thousands of Wikipedia-like answers, (2) calibration of the QA model's confidence scores, and (3) sequential decision-making that incorporates knowledge of the QA model, its calibration, and what the opponent may do. We make two contributions: (1) collecting and curating a large factoid QA dataset and an accompanying gameplay dataset, and (2) developing a model that addresses these three machine learning challenges. In addition to offline evaluation, we pitted our model against some of the most accomplished trivia players in the world in a series of exhibition matches spanning several years. Throughout this paper, we show that collaborations with the vibrant trivia community have contributed to the quality of our dataset, spawned new research directions, and doubled as an exciting way to engage the public with research in machine learning and natural language processing.
BibTeX
@misc{rodriguez2019quizbowl,
  author = {Pedro Rodriguez and Shi Feng and Mohit Iyyer and He He and Jordan L. Boyd-Graber},
  title = {{Quizbowl: The Case for Incremental Question Answering}},
  year = {2019},
  eprint = {1904.04792},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1904.04792},
}
Figure