DeepBlueAI at TextGraphs 2021 Shared Task: Treating Multi-Hop Inference Explanation Regeneration as A Ranking Problem

Chunguang Pan, Bingyan Song, Zhipeng LuoNAACL 2021

TL;DR

This paper presents DeepBlueAI's winning system for the TextGraphs 2021 shared task on multi-hop inference explanation regeneration. They frame the problem as a ranking task, using pre-trained language models (PLMs) fine-tuned with triplet loss for both initial retrieval and re-ranking of explanatory facts, achieving state-of-the-art results.

What Problem Does This Paper Address?

The core challenge tackled is multi-hop inference explanation regeneration. Imagine a science question with a correct answer. The task is to find a set of explanatory "facts" from a large knowledge base that logically connect the question to the answer, requiring multiple steps of reasoning (hence "multi-hop"). This is crucial for building more transparent and understandable AI systems, especially in question answering. The specific problem is to rank candidate explanatory facts for their relevance and completeness in explaining the answer.

Methods

The DeepBlueAI system employs a two-stage approach:

  1. Retrieval: They use pre-trained language models (PLMs) like RoBERTa and ERNIE, fine-tuned with a triplet loss function. The triplet loss aims to minimize the distance between an "anchor" (the question-answer pair) and a "positive" (a relevant explanatory fact), while maximizing the distance to a "negative" (an irrelevant fact). This helps the model learn to identify relevant facts. They experimented with different strategies for selecting negative samples, finding that using facts from the same table file as the positive sample was most effective.
  2. Re-ranking: The top-K candidates retrieved in the first stage are then re-ranked using the same model architecture. This second pass further refines the selection of the most pertinent explanations.

To boost performance, they implemented ensembling, averaging the results from models trained with different PLMs and random seeds.

Datasets

The primary dataset used is derived from the AI2 Reasoning Challenge (ARC) dataset, augmented with approximately 250,000 pre-release expert-generated relevancy ratings. The knowledge base supporting these questions and explanations contains about 9,000 facts, encompassing scientific and common-sense knowledge.

Key Insights and Ideas

  • Framing as a Ranking Problem: Treating explanation regeneration as a ranking task, rather than a simple classification or generation problem, proved highly effective. This aligns well with evaluation metrics like NDCG.
  • Triplet Loss for Relevance: The triplet loss is a powerful tool for learning embeddings that capture the semantic relationship between question-answer pairs and explanatory facts, enabling effective retrieval of relevant information.
  • Negative Sample Selection Matters: The strategy for choosing negative samples significantly impacts performance. Using contextually similar but irrelevant facts (from the same table file) proved more beneficial than random negatives.
  • Ensembling for Robustness: Averaging predictions from multiple models (different PLMs, seeds) leads to more robust and higher-performing systems, a common but effective technique in competitive NLP tasks.

Key Results

DeepBlueAI's system achieved the first place in the TextGraphs 2021 shared task. Their system outperformed the second-best system by 4.93 points in terms of NDCG. The paper also shows that their ensemble retriever achieved an NDCG score of 97.56% on the test set, significantly outperforming a TF-IDF baseline. The re-ranking stage further improved performance, with their ensemble re-ranker achieving an NDCG@100 of 0.8171.

Limitations and Future Work

The paper doesn't explicitly detail limitations or future work beyond the general statement that their approach proved effective. However, one could infer that the reliance on pre-trained language models and the computational cost of fine-tuning and ensembling might be areas for future optimization. Further exploration into more sophisticated negative sampling strategies or novel architectures for multi-hop reasoning could also be promising directions.

Summary updated

BibTeX
@inproceedings{pan2021deepblueai,
  author = {Chunguang Pan and Bingyan Song and Zhipeng Luo},
  title = {{DeepBlueAI at TextGraphs 2021 Shared Task: Treating Multi-Hop Inference Explanation Regeneration as A Ranking Problem}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2021},
}
Figure