Building a Buzzer-quiz Answering System

Naoya Sugiura, Kosuke Yamada, Ryohei Sasano, Koichi Takeda, Katsuhiko ToyamaACL 2023

TL;DR

This paper introduces the first systems designed to answer "buzzer quiz" questions, which are answered before the question is fully read. The authors propose two models, a GPT-only system and a GPT+DPR system, and crucially, methods to estimate the confidence of their answers. This work is significant because it tackles the challenge of answering incomplete questions, a crucial aspect of real-time human-like conversational AI.

What Problem Does This Paper Address?

Traditional question-answering (QA) systems are trained on complete questions and expect all necessary information to be present. However, buzzer quizzes present a unique challenge: contestants must answer questions while they are being read aloud, often before the question is fully articulated. This means systems need to predict answers from partial information and, critically, assess the likelihood of their prediction being correct to avoid penalties for wrong answers. Existing QA datasets like Natural Questions and TriviaQA don't capture this dynamic, making them unsuitable for building buzzer quiz systems. The paper addresses this gap by developing systems that can handle incomplete questions and by proposing a confidence score mechanism to gauge answer accuracy in real-time.

Methods

The authors propose two distinct buzzer-quiz answering systems:

  • GPT-only System: This approach directly uses a pre-trained autoregressive language model (GPT) to generate an answer from a partial question. The input format is designed to prompt the model to complete the question and then provide the answer.
  • GPT+DPR System: This system first uses GPT to "complete" the incomplete question by inferring the missing parts. The completed question is then fed into a retriever-reader model (based on Dense Passage Retrieval - DPR) to find relevant documents and extract the answer.

A key contribution is the method for estimating answer confidence:

  • GPT-only Confidence: The confidence score is derived from the generation probability of the first token in the predicted answer. This is because the initial token often sets the direction of the answer.
  • GPT+DPR Confidence: For this system, three internal scores are considered: the document score (how relevant a document is), the extraction score (how likely a span within a document is the answer), and their average.

Datasets

The study utilizes two datasets:

  • AIO Official Dataset: This dataset contains past questions from Japanese quiz competitions and was used for training and evaluation. It was split into training, development, and test sets.
  • Minhaya Dataset: This dataset, from a Japanese quiz application, was used as additional training data.

Notably, the DPR model required positive and negative documents for training, which were derived from the AIO dataset. The Minhaya dataset was used exclusively for training the GPT model.

Key Insights and Ideas

  • The Importance of Confidence: The paper strongly emphasizes that simply predicting an answer isn't enough for buzzer quizzes. Knowing how likely an answer is correct is paramount to avoid penalties, mirroring human strategy in such games.
  • Incomplete Questions are Fundamentally Different: The authors highlight that answering incomplete questions is a distinct challenge from answering complete ones, often involving ambiguity and a higher risk of incorrect predictions.
  • GPT-only Shines in Early Stages: Surprisingly, the GPT-only model often exhibited higher accuracy than the GPT+DPR model when the answer generation rate was low (i.e., when the system was more selective about answering). This suggests its ability to make educated guesses even with very little information.
  • Confidence Scores are Validated: The research successfully demonstrates that internal model scores can be effectively used as confidence indicators, correlating with answer accuracy.

Key Results

  • Accuracy: As question completeness decreased, accuracy for both models decreased. However, the decline was gentler from 100% to 75% completeness, suggesting crucial information is often in the first half of questions. GPT+DPR slightly outperformed GPT-only at 100% completeness, but performance was similar for incomplete questions.
  • Confidence Score Effectiveness: The study showed that confidence scores help improve accuracy by filtering out low-confidence answers. The generation score for GPT-only and the document score for GPT+DPR yielded the highest AUC values, confirming their effectiveness.
  • GPT-only's Advantage in Low Answer Rates: Figure 1 illustrates that when the answer generation rate was low (e.g., less than 0.8), the GPT-only model achieved higher accuracy, particularly at low question completeness levels (e.g., 25%). This suggests it's more robust when making fewer, more confident predictions.

Limitations and Future Work

  • Real-time Response: The current systems do not account for the time taken to respond, which is critical for actual buzzer quizzes. They also cannot generate real-time responses.
  • Language Specificity: Experiments were conducted solely in Japanese. The authors acknowledge that English, with its different sentence structure, might yield different results, requiring further investigation.
  • Future Directions: The authors suggest exploring more powerful models like FiD or GPT-4 to improve accuracy. They also aim to validate the performance differences between their systems and human players.

Summary updated

BibTeX
@inproceedings{sugiura2023building,
  author = {Naoya Sugiura and Kosuke Yamada and Ryohei Sasano and Koichi Takeda and Katsuhiko Toyama},
  title = {{Building a Buzzer-quiz Answering System}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
}
Figure