Abstract
Understanding qualitative relationships—like knowing that heavier cars accelerate slower—is fundamental to human intelligence. We introduce QUARTZ, the first open-domain dataset for textual qualitative reasoning. By pairing 3,864 multiple-choice questions with 400 general knowledge sentences, QUARTZ challenges models to apply arbitrary qualitative relationships to novel scenarios. State-of-the-art BERT models fall more than 20% behind human performance, highlighting a significant gap in current machine reading comprehension.
Introduction
While NLP has advanced in general Question Answering (QA), reasoning about textual qualitative knowledge (e.g., "more X leads to less Y") remains a challenge. Previous datasets like QuaRel were limited to small, fixed ontologies. QUARTZ removes these constraints, requiring models to dynamically comprehend and reason over arbitrary qualitative statements found in text.
The QUARTZ Dataset
The QUARTZ dataset consists of 3,864 situated multiple-choice questions grounded in a background corpus of 400 qualitative knowledge sentences. To prevent models from relying on simple word-matching heuristics, the dataset is balanced with "flipped" questions that invert comparatives (e.g., changing "more" to "less").
The dataset also includes rich annotations of the specific properties being compared and how they manifest linguistically.
Methodology
The authors evaluated several BERT-large configurations:
- Standard Fine-Tuning: Training directly on QUARTZ.
- Information Retrieval (IR): Using the question to retrieve the relevant knowledge sentence from the 400-sentence corpus before feeding both into BERT.
- Pre-Fine-Tuning (PFT): Fine-tuning BERT on a large-scale reading comprehension dataset (RACE) before training on QUARTZ.
Key Insights
- The Necessity of Flipping: Balancing training data with flipped logic forces models to learn qualitative relationships rather than shallow word-matching.
- The Power of Pre-Fine-Tuning: Pre-training on RACE teaches general QA mechanics, significantly improving performance on the specific qualitative task.
- Latent Knowledge: BERT achieves 68.8% accuracy without background knowledge, suggesting it internalizes some commonsense qualitative relationships during pre-training.
- Semantic Gaps: The primary hurdle is the linguistic disconnect between general knowledge (e.g., "increased altitude") and situated questions (e.g., "higher").
Results and Analysis
The best model (BERT-PFT with IR) achieved 73.7% accuracy, significantly trailing the human baseline of 95.0%. Even with "gold" knowledge sentences provided, the model only reached 79.8%, indicating that reasoning, not just retrieval, is the bottleneck. Furthermore, models fine-tuned on general science questions (BERT-Sci) performed poorly (54.6%), suggesting qualitative reasoning is a distinct skill.
Conclusion
QUARTZ reveals a substantial gap in the ability of current NLP models to perform open-domain qualitative reasoning. Future work should focus on scaling background corpora and improving structured reasoning to handle complex linguistic variations and chained influences.
Summary updated