Red Dragon AI at TextGraphs 2021 Shared Task: Multi-Hop Inference Explanation Regeneration by Matching Expert Ratings

Vivek Kalyan, Sam Witteveen, Martin Andrews

Abstract: Creating explanations for answers to science questions is a challenging task that requires multi-hop inference over a large set of fact sentences. This year, to refocus the Textgraphs Shared Task on the problem of gathering relevant statements (rather than solely finding a single 'correct path'), the WorldTree dataset was augmented with expert ratings of 'relevance' of statements to each overall explanation. Our system, which achieved second place on the Shared Task leaderboard, combines initial statement retrieval; language models trained to predict the relevance scores; and ensembling of a number of the resulting rankings. Our code implementation is made available at https://github.com/mdda/worldtree_corpus/tree/textgraphs_2021

Introduction

Answering complex science questions often requires "multi-hop inference"—the ability to connect multiple disparate facts to form a coherent explanation. Historically, benchmarks like WorldTree required models to find the exact "gold path" of facts used by a human annotator. However, this is notoriously difficult due to semantic drift, where the model wanders off-topic as it hops between facts, and the high branching factor of possible explanations.

The 2021 TextGraphs Shared Task refocused the problem: instead of finding one specific path, models were tasked with ranking a large set of facts based on their "relevance" to an explanation, as determined by expert ratings. The challenge lies in efficiently filtering thousands of facts and then precisely scoring the most promising candidates to match human judgment.

Methodology

The Red Dragon AI system implements a three-stage pipeline designed to balance broad recall with high-precision ranking.

1. Iterative BM25 (I-BM25) Retrieval

To handle the massive search space of the WorldTree corpus, we developed a modified version of BM25. Unlike standard retrieval, I-BM25 is iterative: it selects the top n candidates, aggregates their vectors to update the query, and repeats. This process emulates multi-hop retrieval by "expanding" the query with context from retrieved facts. We tuned this stage to retrieve a "shortlist" of 200 statements. As shown in Figure 1, this size provides a strong balance, capturing approximately 93.8% of relevant facts while maintaining computational efficiency for the subsequent neural stages.

Recall of ground truth statements across different expert rating thresholds and retrieval set sizes.

2. BERT-based Regression

The core of the scoring system is a BERT model fine-tuned as a regressor rather than a classifier. The model processes a concatenated string of [Question + Answer] and [Fact] to predict a floating-point "Expert Rating." This "early-interaction" approach allows the model to look deeply at the relationship between the question and the candidate fact. By treating relevance as a continuous score, the model captures the nuances of human expert ratings more effectively than binary classification.

3. Ensembling

To maximize performance, we ensembled four BERT models (trained with different seeds) and one SciBERT model pre-trained on scientific text. We utilized a simple linear combination of the ranks from these models to produce the final submission.

Key Results and Analysis

The system achieved 2nd place overall in the TextGraphs-15 Shared Task. The progression of performance across different model configurations is detailed in Figure 3.

NDCG scores for baseline and proposed models on development and test sets.

Our best ensemble achieved an NDCG of 0.7705 on the hidden test set. Ablation studies revealed that moving from a base I-BM25 (0.6583) to a single BERT re-ranker (0.7580) provided the largest performance jump, while ensembling added a final ~1.2% boost.

We also calculated an "Oracle" score—the maximum possible score if our re-ranker was perfect given the facts retrieved by I-BM25. This analysis showed that our retrieval stage provided a high ceiling (0.9378), confirming that the bottleneck for further improvement lies primarily in the re-ranking stage rather than initial retrieval.

Discussion and Future Work

While the system performed strongly, several "negative results" highlight the task's difficulty. For instance, attempting a two-stage model (predicting binary relevance before the specific rating) did not improve scores. Furthermore, while large language models excel at local relevance and common sense, they still struggle to grasp the "bigger picture" of a multi-step logical argument.

Future work should explore methods to better incorporate the structured, graph-like information present in the WorldTree corpus. Moving beyond local fact-matching to global explanation coherence remains the primary challenge for multi-hop inference systems.

Summary updated

Abstract

Creating explanations for answers to science questions is a challenging task that requires multi-hop inference over a large set of fact sentences. This year, to refocus the Textgraphs Shared Task on the problem of gathering relevant statements (rather than solely finding a single 'correct path'), the WorldTree dataset was augmented with expert ratings of 'relevance' of statements to each overall explanation. Our system, which achieved second place on the Shared Task leaderboard, combines initial statement retrieval; language models trained to predict the relevance scores; and ensembling of a number of the resulting rankings. Our code implementation is made available at https://github.com/mdda/worldtree_corpus/tree/textgraphs_2021
BibTeX
@inproceedings{kalyan2021red,
  author = {Vivek Kalyan and Sam Witteveen and Martin Andrews},
  title = {{Red Dragon AI at TextGraphs 2021 Shared Task: Multi-Hop Inference Explanation Regeneration by Matching Expert Ratings}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2021},
}
Figure