Abstract
The Baidu PGL team achieved first place in the TextGraphs 2020 Shared Task on Explanation Regeneration. The task requires selecting facts from a large knowledge base to explain elementary science questions. Our system employs a three-stage pipeline: retrieval using a pre-trained ERNIE 2.0 model, pointwise re-ranking, and a final graph-based ranking using a novel "ERNIESage" architecture. By combining the semantic understanding of transformers with the structural reasoning of Graph Neural Networks (GNNs), our system outperformed the runner-up by 1.90 points, achieving a MAP of 60.33%.
The Challenge of Multi-hop Inference
Explanation regeneration involves selecting a set of supporting facts (typically 4-6 sentences) from a knowledge base to form a coherent reasoning chain for a given question and its correct answer. The primary difficulty lies in multi-hop inference, where "gold" explanatory facts may have little to no lexical overlap with the question itself.
Standard information retrieval (IR) methods often fail to capture these "bridge" facts. For instance, explaining why a block of ice has mass requires connecting "ice" to "object" and "object" to "mass."
Methodology: The PGL Pipeline
Our approach utilizes a three-stage pipeline to narrow down thousands of facts into a precise, ranked list.
1. Retrieval and Re-ranking
We first use ERNIE 2.0 as a retriever to score the relevance of candidate facts by concatenating the question, answer, and fact. This stage identifies the top 100 candidate facts with high recall (92.03% Oracle MAP@top100). A second ERNIE-based model then performs pointwise re-ranking to further refine this list.
2. The ERNIESage Architecture
To capture the inter-dependencies between facts, we construct a graph where facts are nodes. Unlike traditional GNNs that use simple aggregation functions, we developed ERNIESage. This architecture applies a transformer to the concatenated tokens of a "center" fact and its "neighbor" facts, enabling granular, token-level interaction during graph convolution.
3. Graph Pruning and Regularization
To reduce noise, we use a pairwise binary classifier to predict relevance between facts before edge construction. We also employ Virtual Adversarial Training (VAT) to improve model robustness against input perturbations.
Results and Discussion
The system was evaluated on the WorldTree V2 corpus, comprising approximately 4,400 science questions. The integration of the GNN component provided a significant boost, raising the MAP from 59.16% to 59.99% on the development set. On the official leaderboard, our model secured first place with a MAP of 60.33%, demonstrating the effectiveness of combining transformers with structural graph learning.
Key insights from our work include:
- Transformers + GNNs: While transformers excel at single-fact relevance, GNNs are essential for multi-hop path construction.
- Token-Level Interaction: Granular interaction between node tokens is superior to using static sentence embeddings.
- Input Optimization: Focusing strictly on the question and correct answer, rather than including distractor options, improves performance.
Conclusion
Our winning entry at TextGraphs 2020 highlights the power of hybrid architectures. By leveraging ERNIESage for deep semantic interaction across a curated graph of facts, we successfully addressed the multi-hop inference challenge in explanation regeneration. Future work will focus on managing graph density and reducing the computational overhead of these large-scale models.
Summary updated