Cheater's Bowl: Human vs. Computer Search Strategies for Open-Domain QA

Wanrong He, Andrew Mao, Jordan L. Boyd-GraberEMNLP 2022 · DOI

Introduction

In the realm of open-domain question answering (QA), systems are tasked with navigating vast knowledge bases like Wikipedia to find precise answers to complex, multi-hop questions. While modern AI models have made significant strides, a substantial performance gap remains between machine retrievers and human experts. This research, presented at EMNLP 2022, investigates the underlying strategies that make humans "virtuosic" searchers and explores how these insights can be used to improve AI performance.

The Cheater’s Bowl Framework

To quantify the differences between human and machine search behaviors, researchers developed Cheater’s Bowl, a gamified web interface. In this environment, trivia experts were challenged to answer difficult questions from datasets like Quizbowl and HotpotQA using a suite of search tools.

The Cheater's Bowl interface allows users to search Wikipedia, highlight evidence, and submit answers.

The system recorded every interaction, including the specific queries written, documents selected, and evidence highlighted. This allowed the researchers to construct formal "action paths" for humans, which could then be compared against the paths taken by state-of-the-art multi-hop models such as GOLDEN retriever and IRRR.

Human vs. Machine: Key Strategic Differences

The study revealed several fundamental ways in which human search strategies diverge from current AI approaches:

1. Precision and Linguistic Composition

Humans prioritize precision over recall. While AI models often generate long, descriptive queries (averaging 12.76 words) that mirror the original question, humans use significantly shorter queries (averaging 2.67 words). These human queries are heavily weighted toward proper nouns and specialized terms.

Comparison of Part-of-Speech (POS) tags in queries shows that humans rely much more heavily on nouns than AI models.

2. Leveraging External Knowledge

A striking 16.3% of human queries included terms found in neither the question nor the retrieved evidence. Humans actively inject "world knowledge" to narrow their search—for instance, adding the term "chemist" to a search for a specific scientist mentioned in a clue.

3. Dynamic Abandonment and Verification

Humans are adept at recognizing unproductive search paths. If a query returns irrelevant results, a human will immediately pivot or start a fresh search. In contrast, many AI models are architecturally "compelled" to select a document from every search result, which can lead to a cycle of "garbage in, garbage out." Furthermore, humans often perform a "verification" step, querying their candidate answer to ensure it logically connects back to the original question.

Despite the recent focus on neural Dense Passage Retrieval (DPR), human experts used traditional keyword-based BM25 for 87% of their queries. They found keyword matching more predictable and effective for locating specific Wikipedia pages they suspected contained the answer.

Results and Impact

The value of human-like intuition was demonstrated through an intervention study. When the IRRR model was "initialized" with human-generated search steps, its accuracy on the most challenging questions jumped from 27.69% to 37.95%—a 10.26% absolute improvement. This suggests that the primary bottleneck for current models is not their ability to process information, but their strategy for finding it.

Future Directions

The authors propose three design principles for the next generation of QA systems:

  • Retriever-Aware Querying: Models should learn to interact dynamically with search engines, refining or abandoning queries based on result quality.
  • Common Sense Injection: Retrievers should move beyond simple substring matching and learn to incorporate external world knowledge.
  • Iterative Verification: Implementing a "check your work" step where models verify candidate answers against the original query.

Summary updated

BibTeX
@inproceedings{he2022cheater,
  author = {Wanrong He and Andrew Mao and Jordan L. Boyd-Graber},
  title = {{Cheater's Bowl: Human vs. Computer Search Strategies for Open-Domain QA}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2022},
  doi = {10.18653/v1/2022.findings-emnlp.266},
}
Figure