Introduction
In the realm of open-domain question answering (QA), systems are tasked with navigating vast knowledge bases like Wikipedia to find precise answers to complex, multi-hop questions. While modern AI models have made significant strides, a substantial performance gap remains between machine retrievers and human experts. This research, presented at EMNLP 2022, investigates the underlying strategies that make humans "virtuosic" searchers and explores how these insights can be used to improve AI performance.
The Cheater’s Bowl Framework
To quantify the differences between human and machine search behaviors, researchers developed Cheater’s Bowl, a gamified web interface. In this environment, trivia experts were challenged to answer difficult questions from datasets like Quizbowl and HotpotQA using a suite of search tools.
The system recorded every interaction, including the specific queries written, documents selected, and evidence highlighted. This allowed the researchers to construct formal "action paths" for humans, which could then be compared against the paths taken by state-of-the-art multi-hop models such as GOLDEN retriever and IRRR.
Human vs. Machine: Key Strategic Differences
The study revealed several fundamental ways in which human search strategies diverge from current AI approaches:
1. Precision and Linguistic Composition
Humans prioritize precision over recall. While AI models often generate long, descriptive queries (averaging 12.76 words) that mirror the original question, humans use significantly shorter queries (averaging 2.67 words). These human queries are heavily weighted toward proper nouns and specialized terms.
2. Leveraging External Knowledge
A striking 16.3% of human queries included terms found in neither the question nor the retrieved evidence. Humans actively inject "world knowledge" to narrow their search—for instance, adding the term "chemist" to a search for a specific scientist mentioned in a clue.
3. Dynamic Abandonment and Verification
Humans are adept at recognizing unproductive search paths. If a query returns irrelevant results, a human will immediately pivot or start a fresh search. In contrast, many AI models are architecturally "compelled" to select a document from every search result, which can lead to a cycle of "garbage in, garbage out." Furthermore, humans often perform a "verification" step, querying their candidate answer to ensure it logically connects back to the original question.
4. Preference for Keyword Search
Despite the recent focus on neural Dense Passage Retrieval (DPR), human experts used traditional keyword-based BM25 for 87% of their queries. They found keyword matching more predictable and effective for locating specific Wikipedia pages they suspected contained the answer.
Results and Impact
The value of human-like intuition was demonstrated through an intervention study. When the IRRR model was "initialized" with human-generated search steps, its accuracy on the most challenging questions jumped from 27.69% to 37.95%—a 10.26% absolute improvement. This suggests that the primary bottleneck for current models is not their ability to process information, but their strategy for finding it.
Future Directions
The authors propose three design principles for the next generation of QA systems:
- Retriever-Aware Querying: Models should learn to interact dynamically with search engines, refining or abandoning queries based on result quality.
- Common Sense Injection: Retrievers should move beyond simple substring matching and learn to incorporate external world knowledge.
- Iterative Verification: Implementing a "check your work" step where models verify candidate answers against the original query.
Summary updated