Abstract
We introduce RoMQA, the first benchmark for robust, multi-evidence, multi-answer question answering (QA). RoMQA contains clusters of questions that are derived from related constraints mined from the Wikidata knowledge graph. RoMQA evaluates robustness of QA models to varying constraints by measuring worst-case performance within each question cluster. Compared to prior QA datasets, RoMQA has more human-written questions that require reasoning over more evidence text and have, on average, many more correct answers. In addition, human annotators rate RoMQA questions as more natural or likely to be asked by people. We evaluate state-of-the-art large language models in zero-shot, few-shot, and fine-tuning settings, and find that RoMQA is challenging: zero-shot and few-shot models perform similarly to naive baselines, while supervised retrieval methods perform well below gold evidence upper bounds. Moreover, existing models are not robust to variations in question constraints, but can be made more robust by tuning on clusters of related questions. Our results show that RoMQA is a challenging benchmark for large language models, and provides a quantifiable test to build more robust QA methods.
Introduction
Most current Question Answering (QA) models exhibit significant brittleness. While a model might correctly identify "Which pianists were born in Paris?", its performance often degrades when constraints are slightly modified—for instance, asking for pianists not born in Paris or specifically classical pianists. This lack of compositional understanding highlights a "robustness gap" in modern Large Language Models (LLMs).
Existing benchmarks typically fail to address three critical areas:
- Robustness Testing: Models are rarely evaluated on clusters of related questions to ensure consistent understanding.
- Answer Complexity: Most datasets focus on single-answer queries, whereas real-world information needs often involve dozens or hundreds of valid entities.
- Evidence Fragmentation: Complex questions frequently require synthesizing facts scattered across multiple documents, a task known as multi-evidence reasoning.
Methods and Human Evaluation
The RoMQA benchmark was developed using a semi-automated pipeline that combines structured knowledge with human creativity. Researchers first mined subject-relation-object triples from Wikidata and grouped related constraints into clusters. Logical queries were then generated using these constraints, incorporating operations like negation (NOT) and conjunction (AND).
To ensure the benchmark reflects real-world language, Amazon Mechanical Turk workers translated these logical forms into natural, human-written questions. Human evaluation confirms that RoMQA questions are perceived as more natural and likely to be asked by people compared to existing datasets like HotpotQA or CWQ.
Dataset Characteristics
RoMQA consists of approximately 29,000 questions (11k train, 7k dev, 11k test). It is distinguished by its scale and complexity compared to datasets like HotpotQA (multi-hop), ComplexWebQuestions (CWQ), and QAMParI (multi-answer).
As shown in the distributions below, RoMQA questions have a significantly higher mean number of answers (108.6) and require substantially more evidence text (averaging 41.6 pieces) to be fully answered.
Key Results and Insights
The evaluation of RoMQA reveals several critical findings:
- The Robustness Gap: There is a massive drop in performance when moving from standard $F_1$ scores to robustness scores (defined by worst-case performance within a question cluster). For top retrieval models, $F_1$ reached 63.8, but the robustness score plummeted to 37.9.
- Retrieval Bottleneck: Models achieve near-perfect performance (95% $F_1$) when provided with "gold" evidence, but performance collapses when they must retrieve evidence themselves from a large corpus.
- Logical Challenges: Models struggle disproportionately with negative constraints. Furthermore, zero-shot and few-shot models (including GPT-3 and OPT-175B) often fail to outperform naive baselines in closed-set evaluations.
- Open Setting Difficulty: In settings where no candidate answers are provided, even the best 8-shot GPT-3 model achieved only 12.6 Precision@10, highlighting the difficulty of generating comprehensive answer lists from memory or retrieval.
Conclusion
RoMQA demonstrates that current LLMs lack true compositional robustness and struggle with the scale of evidence required for complex, multi-answer questions. Future research should focus on developing retrievers capable of set-based logic and architectures that move beyond pattern matching toward true logical reasoning.
Summary updated