Abstract: We present a new question set, text corpus, and baselines assembled to encourage AI research in advanced question answering. Together, these constitute the AI2 Reasoning Challenge (ARC), which requires far more powerful knowledge and reasoning than previous challenges such as SQuAD or SNLI. The ARC question set is partitioned into a Challenge Set and an Easy Set, where the Challenge Set contains only questions answered incorrectly by both a retrieval-based algorithm and a word co-occurence algorithm. The dataset contains only natural, grade-school science questions (authored for human tests), and is the largest public-domain set of this kind (7,787 questions). We test several baselines on the Challenge Set, including leading neural models from the SQuAD and SNLI tasks, and find that none are able to significantly outperform a random baseline, reflecting the difficult nature of this task. We are also releasing the ARC Corpus, a corpus of 14M science sentences relevant to the task, and implementations of the three neural baseline models tested. Can your model perform better? We pose ARC as a challenge to the community.
Introduction
In recent years, the AI community has celebrated massive leaps in Question Answering (QA) performance on benchmarks like SQuAD and SNLI. However, these successes may be deceptive. Many existing datasets allow models to succeed by using "surface-level cues"—essentially pattern matching or word-overlap tricks—rather than actual reasoning or a deep understanding of the world.
If a model can find the answer to a question simply by identifying a sentence in a corpus that shares the most words with the query, it is not truly reasoning. The AI2 Reasoning Challenge (ARC) aims to move the goalposts toward "Advanced QA," which requires combining multiple facts, using commonsense knowledge, and handling complex linguistic structures that simple retrieval-based algorithms cannot navigate.
The ARC Dataset
The ARC dataset consists of 7,787 multiple-choice science questions spanning grades 3 through 9. To ensure the benchmark specifically targets reasoning, the researchers used a functional filtering process. They ran every question through two baseline solvers: an Information Retrieval (IR) solver and a Pointwise Mutual Information (PMI) solver. Questions that both simple solvers failed to answer correctly were moved into the Challenge Set, while the remainder formed the Easy Set.
These questions are not just difficult due to linguistic complexity; they cover a diverse array of knowledge types, including definitions, structural descriptions, and causal processes. This variety ensures that models cannot rely on a single type of logic to succeed.
Accompanying the questions is the ARC Corpus, a massive collection of 14 million science-related sentences (1.4GB of text) scraped from the web. This provides the necessary knowledge base for models to attempt these questions.
Key Insights and Results
Analysis of the ARC Corpus suggests it contains the information needed to answer approximately 95% of the Challenge questions. Therefore, the failure of current models is not due to a lack of data, but a lack of reasoning—the ability to connect the dots between disparate facts.
In the Challenge Set, incorrect answers are often "distractors" that are highly correlated with the question words (e.g., a question about "minerals" might have "hardness" as an incorrect but statistically likely choice). This specifically trips up models that rely on statistical word associations. The results of baseline testing are a sobering reality check for the field:
- Easy Set: Most baseline models performed reasonably well, with scores ranging from 58% to 77%.
- Challenge Set: Every tested model—including advanced neural networks like BiDAF and DecompAttn—hit a brick wall. Most scored between 24% and 27%, barely outperforming a random guesser (25%).
- Top Performer: The Decomposed Graph Entailment Model (DGEM) performed best on the Challenge Set at 27.11%, but this remains only marginally better than chance.
Conclusion and Future Work
The ARC benchmark highlights a major bottleneck in current NLP: multi-step reasoning. Most current models are "one-shot" learners that look for a single sentence providing the answer. ARC questions often require "chaining" facts (e.g., Fact A + Fact B = Answer). Future research must focus on models that can perform this kind of multi-fact integration and better incorporate structured commonsense knowledge to bridge the gap between simple retrieval and true machine intelligence.
Summary updated