RoMQA: A Benchmark for Robust, Multi-evidence, Multi-answer Question Answering

Victor Zhong, Weijia Shi, Wen-tau Yih, Luke Zettlemoyer

Abstract

We introduce RoMQA, the first benchmark for robust, multi-evidence, multi-answer question answering (QA). RoMQA contains clusters of questions that are derived from related constraints mined from the Wikidata knowledge graph. RoMQA evaluates robustness of QA models to varying constraints by measuring worst-case performance within each question cluster. Compared to prior QA datasets, RoMQA has more human-written questions that require reasoning over more evidence text and have, on average, many more correct answers. In addition, human annotators rate RoMQA questions as more natural or likely to be asked by people. We evaluate state-of-the-art large language models in zero-shot, few-shot, and fine-tuning settings, and find that RoMQA is challenging: zero-shot and few-shot models perform similarly to naive baselines, while supervised retrieval methods perform well below gold evidence upper bounds. Moreover, existing models are not robust to variations in question constraints, but can be made more robust by tuning on clusters of related questions. Our results show that RoMQA is a challenging benchmark for large language models, and provides a quantifiable test to build more robust QA methods.

Introduction: The Challenge of Robustness

Most current question-answering (QA) benchmarks evaluate a model's ability to find a single correct answer to a static query. However, real-world information seeking is dynamic and complex. A robust system should not only answer a specific question like "Which pianists born in Paris play Western classical music?" but also handle logical variations, such as broader queries or those involving negations (e.g., "Who was a Western classical pianist not born in Paris?").

Existing compositional QA datasets often fail to evaluate this robustness to constraint variations. Furthermore, many "complex" datasets feel artificial, featuring questions that humans would rarely ask or that can be bypassed using simple heuristics. RoMQA addresses these gaps by providing a natural, challenging, and quantifiable testbed for building models that understand the underlying logic of a query.

Methodology: Building Logical Clusters

RoMQA was constructed using a multi-stage pipeline designed to ensure logical consistency and natural language quality:

  1. Constraint Sampling: Subject-proposition-object triples were mined from Wikidata.
  2. Clustering: Related constraints sharing answer entities were grouped to create clusters of logically related questions.
  3. Logical Query Generation: Combinations of constraints were sampled to form queries involving conjunctions (AND) and negations (NOT).
  4. Human-in-the-loop Annotation: Mechanical Turk workers translated logical queries into conversational English, followed by a verification step to ensure logical alignment.

To measure performance, the authors introduced a Robustness Score, which tracks the worst-case performance within a cluster. This metric penalizes models that can answer an "easy" version of a question but fail when a constraint is modified.

Dataset Characteristics

RoMQA represents a significant scale-up in complexity compared to existing benchmarks. It contains 28,000 human-written questions, a 10x increase over previous multi-answer datasets like QAMParI.

RoMQA features a significantly higher number of answers and required evidence compared to prior benchmarks.

As shown in the distributions above, RoMQA questions typically have a high number of answers (mean of 108.6) and require reasoning over substantial evidence (mean of 41.6 sentences per question), far exceeding the complexity of HotpotQA or CWQ.

Key Insights and Results

The Robustness Gap

Experiments reveal a massive performance drop when moving from standard accuracy to the worst-case robustness metric. A supervised model achieving 63.8 $F_1$ on individual questions saw its performance plummet to 37.9 $F_1$ when evaluated for robustness across clusters. This suggests that current models are often "right for the wrong reasons."

Naturalness and Human Likelihood

One of RoMQA's strengths is the perceived naturalness of its questions. Human evaluators rated RoMQA questions as significantly more likely to be asked by a person compared to older datasets like HotpotQA or CWQ, which often contain overly precise or unusual constraints.

Human naturalness ratings show that RoMQA and QAMParI questions are perceived as more realistic than those in HotpotQA or CWQ.

Model Performance

  • LLM Struggles: Large Language Models (LLMs) like GPT-3 and OPT-175B struggle with set-based reasoning. In zero-shot and few-shot settings, they performed similarly to a naive baseline that predicts all candidates as answers.
  • Supervised Limits: The best supervised "retrieve-then-classify" model (BART-large) reached 63.8 $F_1$, well below the 95.0 $F_1$ achieved when provided with gold evidence. This highlights the difficulty of aggregating evidence across a massive corpus.

Conclusion and Future Work

RoMQA demonstrates that linguistic robustness remains an unsolved challenge. While training on question clusters improves robustness, a large gap remains compared to human-level performance. Future work is needed to improve model efficiency in set-theoretic operations (e.g., handling large negations) and to expand the benchmark's scope beyond English and Wikidata-derived facts.

Summary updated

Abstract

We introduce RoMQA, the first benchmark for robust, multi-evidence, multi-answer question answering (QA). RoMQA contains clusters of questions that are derived from related constraints mined from the Wikidata knowledge graph. RoMQA evaluates robustness of QA models to varying constraints by measuring worst-case performance within each question cluster. Compared to prior QA datasets, RoMQA has more human-written questions that require reasoning over more evidence text and have, on average, many more correct answers. In addition, human annotators rate RoMQA questions as more natural or likely to be asked by people. We evaluate state-of-the-art large language models in zero-shot, few-shot, and fine-tuning settings, and find that RoMQA is challenging: zeroshot and few-shot models perform similarly to naive baselines, while supervised retrieval methods perform well below gold evidence upper bounds. Moreover, existing models are not robust to variations in question constraints, but can be made more robust by tuning on clusters of related questions. Our results show that RoMQA is a challenging benchmark for large language models, and provides a quantifiable test to build more robust QA methods.
BibTeX
@inproceedings{zhong2023romqa,
  author = {Victor Zhong and Weijia Shi and Wen-tau Yih and Luke Zettlemoyer},
  title = {{RoMQA: A Benchmark for Robust, Multi-evidence, Multi-answer Question Answering}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2023},
  url = {https://openreview.net/forum?id=1GUTpLbbhS},
}
Figure