PGL at TextGraphs 2020 Shared Task: Explanation Regeneration using Language and Graph Learning Methods

Weibin Li, Yuxiang Lu, Zhengjie Huang, Weiyue Su, Jiaxiang Liu, Shikun Feng, Yu Sun2020 · DOI

Abstract

The Baidu PGL team achieved first place in the TextGraphs 2020 Shared Task on Explanation Regeneration. The task requires selecting facts from a large knowledge base to explain elementary science questions. Our system employs a three-stage pipeline: retrieval using a pre-trained ERNIE 2.0 model, pointwise re-ranking, and a final graph-based ranking using a novel "ERNIESage" architecture. By combining the semantic understanding of transformers with the structural reasoning of Graph Neural Networks (GNNs), our system outperformed the runner-up by 1.90 points, achieving a MAP of 60.33%.

The Challenge of Multi-hop Inference

Explanation regeneration involves selecting a set of supporting facts (typically 4-6 sentences) from a knowledge base to form a coherent reasoning chain for a given question and its correct answer. The primary difficulty lies in multi-hop inference, where "gold" explanatory facts may have little to no lexical overlap with the question itself.

Figure 1: An example of multi-hop reasoning where "bridge" facts connect disparate concepts like "ice" and "mass" through the intermediate concept of an "object".

Standard information retrieval (IR) methods often fail to capture these "bridge" facts. For instance, explaining why a block of ice has mass requires connecting "ice" to "object" and "object" to "mass."

Methodology: The PGL Pipeline

Our approach utilizes a three-stage pipeline to narrow down thousands of facts into a precise, ranked list.

1. Retrieval and Re-ranking

We first use ERNIE 2.0 as a retriever to score the relevance of candidate facts by concatenating the question, answer, and fact. This stage identifies the top 100 candidate facts with high recall (92.03% Oracle MAP@top100). A second ERNIE-based model then performs pointwise re-ranking to further refine this list.

2. The ERNIESage Architecture

To capture the inter-dependencies between facts, we construct a graph where facts are nodes. Unlike traditional GNNs that use simple aggregation functions, we developed ERNIESage. This architecture applies a transformer to the concatenated tokens of a "center" fact and its "neighbor" facts, enabling granular, token-level interaction during graph convolution.

Figure 2: The ERNIESage architecture, which integrates ERNIE blocks within a GNN framework to perform token-level interaction during neighbor aggregation.

3. Graph Pruning and Regularization

To reduce noise, we use a pairwise binary classifier to predict relevance between facts before edge construction. We also employ Virtual Adversarial Training (VAT) to improve model robustness against input perturbations.

Results and Discussion

The system was evaluated on the WorldTree V2 corpus, comprising approximately 4,400 science questions. The integration of the GNN component provided a significant boost, raising the MAP from 59.16% to 59.99% on the development set. On the official leaderboard, our model secured first place with a MAP of 60.33%, demonstrating the effectiveness of combining transformers with structural graph learning.

Key insights from our work include:

  • Transformers + GNNs: While transformers excel at single-fact relevance, GNNs are essential for multi-hop path construction.
  • Token-Level Interaction: Granular interaction between node tokens is superior to using static sentence embeddings.
  • Input Optimization: Focusing strictly on the question and correct answer, rather than including distractor options, improves performance.

Conclusion

Our winning entry at TextGraphs 2020 highlights the power of hybrid architectures. By leveraging ERNIESage for deep semantic interaction across a curated graph of facts, we successfully addressed the multi-hop inference challenge in explanation regeneration. Future work will focus on managing graph density and reducing the computational overhead of these large-scale models.

Summary updated

Abstract

This paper describes the system designed by the Baidu PGL Team which achieved the first place in the TextGraphs 2020 Shared Task. The task focuses on generating explanations for elementary science questions. Given a question and its corresponding correct answer, we are asked to select the facts that can explain why the answer is correct for that question and answering (QA) from a large knowledge base. To address this problem, we use a pre-trained language model to recall the top-K relevant explanations for each question. Then, we adopt a re-ranking approach based on a pre-trained language model to rank the candidate explanations. To further improve the rankings, we also develop an architecture consisting both powerful pre-trained transformers and GNNs to tackle the multi-hop inference problem. The official evaluation shows that, our system can outperform the second best system by 1.91 points.
BibTeX
@misc{li2020pgl,
  author = {Weibin Li and Yuxiang Lu and Zhengjie Huang and Weiyue Su and Jiaxiang Liu and Shikun Feng and Yu Sun},
  title = {{PGL at TextGraphs 2020 Shared Task: Explanation Regeneration using Language and Graph Learning Methods}},
  year = {2020},
  doi = {10.18653/v1/2020.textgraphs-1.11},
}
Figure