TL;DR
This paper presents a system that achieved second place in the TextGraphs-15 Shared Task, focusing on generating explanations for science questions that require multi-hop reasoning. Their approach combines an iterative retrieval method with fine-tuned BERT language models to predict the relevance of explanation statements, ultimately ensembling these predictions for improved performance.
What Problem Does This Paper Address?
Answering complex science questions often requires piecing together information from multiple sources, a process known as multi-hop inference. Traditional question-answering datasets often limit the complexity by requiring only a few facts to form an answer. The TextGraphs Shared Task, however, uses the WorldTree dataset, which features over 5,000 questions and answers, with "gold" explanations requiring an average of six facts, and sometimes up to sixteen. This year's task shifted focus from finding a single "correct path" to ranking individual explanation statements based on their relevance to the question and answer, using Normalized Discounted Cumulative Gain (NDCG) as the evaluation metric. This change aimed to address the challenge of semantic drift and the difficulty of perfectly matching human-generated explanations.
Methods
The system employs a three-stage pipeline:
- Initial Retrieval (I-BM25): They adapt an iterative BM25 retrieval method. This approach first pre-computes sparse document vectors for all candidate explanations. Then, for each question, it iteratively aggregates vectors of the closest explanation candidates, updating the query vector. This method was chosen for its speed and ability to reduce semantic drift, and was optimized to retrieve 200 statements, balancing recall with downstream processing costs.
- Language Models for Rating (BERT): A BERT language model is fine-tuned as a regression task to predict the "Expert Rating" of each explanation statement. The input to BERT is a concatenation of the question, its correct answer, and a candidate explanation statement, separated by special tokens. The model outputs a floating-point score, which is then used to rank the explanations.
- Ensembling of Rankings: The final stage involves ensembling the ranked outputs from multiple models. Specifically, they ensemble four BERT models (fine-tuned with different random seeds) and one SciBERT model. The ensemble method used is a simple linear combination of the individual model rankings.
Datasets
- WorldTree V2: This dataset, augmented with expert-annotated relevancy ratings for facts, serves as the primary dataset for training and evaluation. It contains over 5,000 questions and answers, with detailed explanations requiring multi-hop inference.
- Dev Set: Used for hyperparameter tuning and model selection.
- Leaderboard: Used for final evaluation of the system's performance on unseen data.
Key Insights and Ideas
- Retrieval as a Strong Baseline: Even conventional information retrieval methods, like their optimized I-BM25, can form a robust foundation for complex reasoning tasks when adapted appropriately.
- BERT's Predictive Power: Fine-tuning BERT for a regression task to predict expert relevance scores proved highly effective, demonstrating the model's ability to understand nuanced relationships between questions, answers, and explanation statements.
- Ensembling for Robustness: Combining the predictions of multiple fine-tuned language models (BERT and SciBERT) significantly boosted performance, suggesting that these models capture complementary aspects of relevance.
- Shift in Task Focus: The change in the Shared Task's scoring metric from finding a single "gold explanation" to ranking relevant statements appears to have favored large language models that can capture "bigger picture" ideas, rather than purely logic-oriented approaches.
Key Results
The system achieved second place on the TextGraphs-15 Shared Task leaderboard.
- The I-BM25 retrieval stage achieved an "Oracle NDCG" of 0.9378, indicating its strong ability to retrieve relevant statements.
- The best performing individual language model was BERT, achieving a Dev NDCG of 0.7679.
- The final ensembled system (I-BM25 + BERT + SciBERT ensemble) achieved a Test NDCG of 0.7705.
Limitations and Future Work
- Two-Stage Representation: An explored two-stage architecture (binary relevance indicator followed by score distribution) did not yield higher scores, possibly due to time constraints for fine-tuning.
- Negative Sampling: Attempts to improve performance by adding more negative samples during training did not result in significant score improvements.
- Ensembling Simplification: While ensembling SciBERT with BERT improved scores, more sophisticated combination methods were considered but suffered from overfitting on the development set.
- Future Work: The authors suggest that future iterations of the Shared Task could benefit from structured information that was previously available in the WorldTree Explanation Corpus v2.1 Desk Reference, allowing for cleaner interpretation of structured table data. They also look forward to further progress on the multi-hop reasoning task.
Summary updated