TextGraphs 2021 Shared Task on Multi-Hop Inference for Explanation Regeneration

Mokanarangan Thayaparan, Marco Valentino, Peter Jansen, Dmitry UstalovNAACL 2021

TL;DR

This paper summarizes the 2021 TextGraphs shared task, which focused on identifying the most relevant facts needed to construct multi-hop explanations for science questions. The challenge lies in sifting through vast knowledge bases to find the crucial pieces of information that form a coherent reasoning chain, a critical step towards more explainable AI.

What Problem Does This Paper Address?

Imagine asking a complex science question, like "Why does the sun appear to move across the sky?" A simple answer like "Earth rotates on its axis" is correct but lacks detail. A truly explanatory answer would involve a chain of facts: Earth's rotation causes an apparent motion of celestial bodies, diurnal motion is this apparent movement, and Earth rotates once per day. This is the essence of multi-hop inference: piecing together multiple facts to form a complete explanation.

Previous work in this area focused on completeness – ensuring all necessary facts were present. However, with increasingly large knowledge bases, a new challenge emerges: relevance. How do we identify the most important facts from a sea of potentially relevant ones? This is crucial because models often struggle with "semantic drift," generating irrelevant or out-of-context information as inference chains grow longer. This shared task specifically targets this relevance problem, aiming to improve the ability of AI systems to select the right facts for building robust explanations.

Methods

The core of the task is framed as a ranking problem. Given a science question and its correct answer, participants' systems must rank facts from a knowledge base. The goal is to place the facts deemed most relevant by human annotators at the top of the ranked list.

The shared task builds upon the WorldTree corpus, which contains science exam questions and their corresponding multi-fact explanations. For the 2021 edition, the dataset was augmented with approximately 250,000 manual relevancy ratings. These ratings are on a 7-point scale (0-6), where 6 indicates the most critical facts for an explanation.

Participating systems employed a variety of techniques, including:

  • Transformer-based language models: Fine-tuned to retrieve relevant facts or re-rank candidate explanations.
  • Graph Neural Networks (GNNs): Used in conjunction with language models.
  • Integer Linear Programming (ILP): For structured reasoning.
  • Sparse retrieval models: Adapted to improve relevance ranking.
  • Ensemble methods: Combining multiple models.

A common strategy involved a two-step process: initial retrieval of a larger set of candidate facts, followed by a re-ranking step to refine the selection.

Datasets

  • WorldTree V2 explanation corpus: This dataset, used in previous iterations, provides science exam questions and multi-fact explanations. It contains 2207 training, 496 development, and 1665 held-out test questions.
  • Augmented relevancy ratings: The 2021 task introduced a large dataset of approximately 250,000 manual relevancy ratings for facts within the WorldTree corpus. These ratings are on a 7-point scale (0-6) and are crucial for evaluating the relevance-ranking aspect of the task.

Key Insights and Ideas

  • Relevance is the new frontier: While completeness was the focus before, the 2021 task highlights that accurately identifying relevant facts is a major bottleneck for building truly useful multi-hop explanations, especially with large knowledge bases.
  • Human ratings are invaluable: The large-scale manual relevancy ratings provided a critical benchmark for evaluating systems on this nuanced aspect of explanation generation.
  • Top systems significantly outperform baselines: The substantial improvement seen in the top-performing systems suggests that current models are capable of learning to identify relevant facts, even if absolute performance is still moderate.
  • Lexical overlap is a double-edged sword: Facts with high lexical overlap with the question/answer are easier to retrieve, but systems still struggle when this overlap is low, indicating a need for deeper semantic understanding.
  • Different knowledge types pose different challenges: Models generally perform better on "retrieval" facts (e.g., taxonomic relations) than on more abstract "inference-supporting" or "complex inference" facts, which require deeper reasoning.

Key Results

The best-performing system, DeepBlueAI, achieved an NDCG score of 0.820. This represents a 32% improvement over the baseline tf.idf model, which scored 0.501. All participating teams substantially improved upon the baseline.

  • DeepBlueAI consistently outperformed other models across various knowledge types and relevance rating bins.
  • Performance generally decreased for more complex types of knowledge (inference-supporting, complex inference) and for facts with lower relevance ratings.
  • Precision@k analysis showed that DeepBlueAI was particularly strong for smaller values of k (up to 10), meaning it ranked the top relevant facts more accurately.

Limitations and Future Work

  • Moderate absolute performance: Despite significant relative improvements, the overall absolute performance on this challenging task is still moderate, indicating substantial room for future work.
  • Struggle with low lexical overlap: Models still show a significant drop in performance when facts have low lexical overlap with the question/answer, suggesting that abstract reasoning and understanding beyond surface-level similarity remains a challenge.
  • Future work: The authors hope that future systems will leverage these findings to build stronger relevancy retrieval subcomponents that can augment compositional inference algorithms for generating more comprehensive and accurate multi-hop explanations. This could involve developing models that better handle abstract scientific knowledge and less direct lexical cues.

Summary updated

BibTeX
@inproceedings{thayaparan2021textgraphs,
  author = {Mokanarangan Thayaparan and Marco Valentino and Peter Jansen and Dmitry Ustalov},
  title = {{TextGraphs 2021 Shared Task on Multi-Hop Inference for Explanation Regeneration}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2021},
}
Figure