Abstract
We introduce RoMQA, the first benchmark for robust, multi-evidence, multi-answer question answering (QA). RoMQA contains clusters of questions that are derived from related constraints mined from the Wikidata knowledge graph. RoMQA evaluates robustness of QA models to varying constraints by measuring worst-case performance within each question cluster. Compared to prior QA datasets, RoMQA has more human-written questions that require reasoning over more evidence text and have, on average, many more correct answers. In addition, human annotators rate RoMQA questions as more natural or likely to be asked by people. We evaluate state-of-the-art large language models in zero-shot, few-shot, and fine-tuning settings, and find that RoMQA is challenging: zero-shot and few-shot models perform similarly to naive baselines, while supervised retrieval methods perform well below gold evidence upper bounds. Moreover, existing models are not robust to variations in question constraints, but can be made more robust by tuning on clusters of related questions. Our results show that RoMQA is a challenging benchmark for large language models, and provides a quantifiable test to build more robust QA methods.
Introduction: The Challenge of Robustness
Most current question-answering (QA) benchmarks evaluate a model's ability to find a single correct answer to a static query. However, real-world information seeking is dynamic and complex. A robust system should not only answer a specific question like "Which pianists born in Paris play Western classical music?" but also handle logical variations, such as broader queries or those involving negations (e.g., "Who was a Western classical pianist not born in Paris?").
Existing compositional QA datasets often fail to evaluate this robustness to constraint variations. Furthermore, many "complex" datasets feel artificial, featuring questions that humans would rarely ask or that can be bypassed using simple heuristics. RoMQA addresses these gaps by providing a natural, challenging, and quantifiable testbed for building models that understand the underlying logic of a query.
Methodology: Building Logical Clusters
RoMQA was constructed using a multi-stage pipeline designed to ensure logical consistency and natural language quality:
- Constraint Sampling: Subject-proposition-object triples were mined from Wikidata.
- Clustering: Related constraints sharing answer entities were grouped to create clusters of logically related questions.
- Logical Query Generation: Combinations of constraints were sampled to form queries involving conjunctions (AND) and negations (NOT).
- Human-in-the-loop Annotation: Mechanical Turk workers translated logical queries into conversational English, followed by a verification step to ensure logical alignment.
To measure performance, the authors introduced a Robustness Score, which tracks the worst-case performance within a cluster. This metric penalizes models that can answer an "easy" version of a question but fail when a constraint is modified.
Dataset Characteristics
RoMQA represents a significant scale-up in complexity compared to existing benchmarks. It contains 28,000 human-written questions, a 10x increase over previous multi-answer datasets like QAMParI.
As shown in the distributions above, RoMQA questions typically have a high number of answers (mean of 108.6) and require reasoning over substantial evidence (mean of 41.6 sentences per question), far exceeding the complexity of HotpotQA or CWQ.
Key Insights and Results
The Robustness Gap
Experiments reveal a massive performance drop when moving from standard accuracy to the worst-case robustness metric. A supervised model achieving 63.8 $F_1$ on individual questions saw its performance plummet to 37.9 $F_1$ when evaluated for robustness across clusters. This suggests that current models are often "right for the wrong reasons."
Naturalness and Human Likelihood
One of RoMQA's strengths is the perceived naturalness of its questions. Human evaluators rated RoMQA questions as significantly more likely to be asked by a person compared to older datasets like HotpotQA or CWQ, which often contain overly precise or unusual constraints.
Model Performance
- LLM Struggles: Large Language Models (LLMs) like GPT-3 and OPT-175B struggle with set-based reasoning. In zero-shot and few-shot settings, they performed similarly to a naive baseline that predicts all candidates as answers.
- Supervised Limits: The best supervised "retrieve-then-classify" model (BART-large) reached 63.8 $F_1$, well below the 95.0 $F_1$ achieved when provided with gold evidence. This highlights the difficulty of aggregating evidence across a massive corpus.
Conclusion and Future Work
RoMQA demonstrates that linguistic robustness remains an unsolved challenge. While training on question clusters improves robustness, a large gap remains compared to human-level performance. Future work is needed to improve model efficiency in set-theoretic operations (e.g., handling large negations) and to expand the benchmark's scope beyond English and Wikidata-derived facts.
Summary updated