Introduction
Semantic parsing—the translation of natural language into formal logical forms like SQL or SPARQL—is often considered the gold standard for answering complex questions. These questions typically involve compositionality (e.g., "Who is the grandson of King David’s father?") or computation (e.g., "Which is the second-highest mountain?"). However, many benchmarks used to evaluate these models contain "factoid" questions that might be easily found in plain text on the web.
This paper challenges the assumption that complex question-answering (QA) benchmarks strictly require deep semantic parsing and structured knowledge bases (KBs). By building WEBQA, a simple model that utilizes only web search snippets, the authors demonstrate that a significant portion of these benchmarks can be solved through text-based redundancy alone, sometimes even outperforming state-of-the-art semantic parsers.
Methodology: The WEBQA Model
The authors developed WEBQA, a log-linear model inspired by traditional web-based QA systems. The pipeline consists of three main stages:
- Candidate Extraction: The question is sent to Google, and the model scrapes the top 100 web snippets. It extracts all 1- to 4-grams as potential answer candidates (approximately 5,000 per question) and filters them down to the top $K$ candidates using a TF-IDF score.
- Candidate Ranking: A log-linear model ranks these candidates based on 892 features.
- Feature Engineering: Key features include match features (using GloVe word embeddings to match context), source features (Google rank of the snippet), and lexical features (entity types). Crucially, the model relies on redundancy: the principle that correct answers appear more frequently across different snippets.
Experimental Setup
The primary evaluation was conducted on the COMPLEXQUESTIONS dataset, which was specifically designed to require multi-step reasoning. To ensure reproducibility, the authors released a version of this dataset that includes the top 100 retrieved web snippets for every question.
Key Results
WEBQA achieved an overall 32.6 F1 score, compared to the state-of-the-art semantic parser (COMPQ) at 40.9 F1. While the overall score is lower, the gap is remarkably small for a model with no access to a structured database.
When performing a "fair" comparison—evaluating only the subset of questions where the correct answer was actually present in the retrieved snippets—WEBQA outperformed the semantic parser, achieving 51.9 F1 compared to COMPQ's 48.5 F1. Ablation studies revealed that TF-IDF was the most critical feature; its removal caused a 12-point drop in F1, highlighting the model's reliance on statistical redundancy.
Analysis and Insights
WEBQA's success stems from its ability to avoid the "Language of the KB" gap. Semantic parsers often struggle to map natural language to idiosyncratic KB predicates (e.g., mapping "wife" to a complex join). WEBQA bypasses this by matching natural language directly to natural language.
However, the model's performance varies significantly by question type. As shown in the analysis above, WEBQA performs well on "conjunction" questions because the web is redundant; it is likely that a single sentence somewhere contains all required entities. Conversely, it struggles with superlatives and relation composition, which require aggregating data or following a chain of facts rarely captured in a single snippet.
Conclusion
WEBQA demonstrates that many "complex" questions can be answered using unstructured text and redundancy. While it lacks the formal reasoning capabilities of semantic parsers for superlatives and multi-step logic, it provides a powerful baseline. The authors suggest that future work should focus on combining web-based QA and semantic parsing, using web redundancy to bridge the gap between natural language and formal knowledge base schemas.
Summary updated