QuaRTz: An Open-Domain Dataset of Qualitative Relationship Questions

Oyvind Tafjord, Matt Gardner, Kevin Lin, Peter Clark

Abstract

We introduce the first open-domain dataset, called QuaRTz, for reasoning about textual qualitative relationships. QuaRTz contains general qualitative statements, e.g., "A sunscreen with a higher SPF protects the skin longer.", twinned with 3864 crowdsourced situated questions, e.g., "Billy is wearing sunscreen with a lower SPF than Lucy. Who will be best protected from the sun?", plus annotations of the properties being compared. Unlike previous datasets, the general knowledge is textual and not tied to a fixed set of relationships, and tests a system's ability to comprehend and apply textual qualitative knowledge in a novel setting. We find state-of-the-art results are substantially (20%) below human performance, presenting an open challenge to the NLP community.

Introduction

Humans navigate the world using qualitative reasoning (QR). We understand that "exercise improves health" and can infer that more time at the gym is likely beneficial. For AI, however, this type of reasoning is notoriously difficult. Most existing QR datasets are "closed-domain," relying on a small, fixed set of properties like "speed" or "mass."

To move beyond these limitations, we present QuaRTz (Qualitative Relationship Test set), an open-domain dataset designed to test if NLP models can apply qualitative logic found in natural language text to novel, situated scenarios. The challenge lies not just in fact retrieval, but in understanding the direction of relationships (more vs. less) and applying them when entities and intensifiers differ from the source text.

The QuaRTz Dataset

The QuaRTz dataset consists of 3,864 crowdsourced questions paired with a background corpus of 400 qualitative sentences. Unlike its predecessor, QuaRel, which was limited to 19 properties, QuaRTz is open-domain and features a vast array of linguistic expressions.

Table 1: Statistics for the QuaRTz dataset, highlighting the balance of "flipped" questions and the distribution of qualitative influences.

The dataset covers three main types of qualitative knowledge: comparatives, commonsense knowledge, and discrete values. These categories test a model's ability to bridge semantic gaps and handle varying levels of abstraction.

Figure 1: Examples of the diverse qualitative relationships captured in QuaRTz, ranging from simple comparatives to complex commonsense knowledge.

Methodology

The researchers at the Allen Institute for AI (AI2) employed a multi-step framework to evaluate machine performance:

  1. Knowledge Extraction: 400 general qualitative sentences were pulled from a large corpus.
  2. Crowdsourced Questioning: Workers created "situated" questions. Crucially, they included "flipped" versions—if one question asks about an increase, the flip asks about a decrease—to prevent models from relying on simple keyword matching.
  3. Modeling: Several BERT-large configurations were tested. The most successful approach utilized Pre-Fine-Tuning (PFT) on the RACE dataset (a large reading comprehension set) before training on QuaRTz, combined with an Information Retrieval (IR) component to locate relevant rules.

Key Insights and Results

The performance gap between humans and machines remains significant. While humans achieve a baseline of 95.0%, the best-performing model (BERT-PFT with IR) reached only 73.7%. Even when provided with the "perfect" knowledge sentence, the model only reached 79.8%, indicating that the bottleneck is the reasoning itself, not just retrieval.

Challenges in Qualitative Reasoning

  • The "Flipping" Trap: Models often fail when logic is reversed. If a model associates "more" with "increase," it struggles when "less" leads to a "decrease."
  • Linguistic Bridging: Systems must recognize that phrases like "warmer" and "increase temperature" are semantically equivalent.
  • Entity Tracking: Models struggle to track properties across multiple entities (e.g., "Jim works out, James doesn't. Who has weaker muscles?"), especially when the question asks for the inverse property.

Conclusion

QuaRTz reveals that while modern Transformers are powerful, they lack the robust qualitative reasoning required for human-like common sense. Future work may involve semantic parsing to map sentences into logical forms or developing architectures capable of handling multi-step "chains" of qualitative influence.

Summary updated

Abstract

We introduce the first open-domain dataset, called QuaRTz, for reasoning about textual qualitative relationships. QuaRTz contains general qualitative statements, e.g., "A sunscreen with a higher SPF protects the skin longer.", twinned with 3864 crowdsourced situated questions, e.g., "Billy is wearing sunscreen with a lower SPF than Lucy. Who will be best protected from the sun?", plus annotations of the properties being compared. Unlike previous datasets, the general knowledge is textual and not tied to a fixed set of relationships, and tests a system's ability to comprehend and apply textual qualitative knowledge in a novel setting. We find state-of-the-art results are substantially (20%) below human performance, presenting an open challenge to the NLP community.
BibTeX
@inproceedings{tafjord2019quartz,
  author = {Oyvind Tafjord and Matt Gardner and Kevin Lin and Peter Clark},
  title = {{QuaRTz: An Open-Domain Dataset of Qualitative Relationship Questions}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2019},
  url = {https://openreview.net/forum?id=Auas3gHd3_B},
}
Figure