TL;DR
This paper introduces ExplanationLP, a novel approach to science question answering that explicitly models both "grounding" (linking abstract concepts to concrete terms) and "abstract" (general scientific knowledge) reasoning chains. By framing question answering as abductive reasoning and using linear programming to find the best explanatory subgraph, ExplanationLP achieves competitive performance with fewer parameters and provides interpretable explanations.
What Problem Does This Paper Address?
Answering complex science questions is a significant challenge in AI, requiring sophisticated reasoning beyond simple pattern matching. While current state-of-the-art models, particularly transformer-based ones, excel at predicting answers, they often act as "black boxes," failing to provide understandable explanations for their predictions. This lack of transparency hinders trust and debugging. Existing explainable approaches often rely on multi-hop graph traversal, which can suffer from "semantic drift" – losing context as the reasoning chain gets longer. This paper aims to address this by developing an explainable AI system that can answer science questions accurately while also generating plausible, interpretable explanations, specifically by modeling the interplay between concrete grounding and abstract scientific knowledge.
Methods
The core of ExplanationLP lies in its abductive reasoning framework, which seeks the "inference to the best explanation." For each candidate answer, the system constructs a hypothesis and then builds a weighted graph of relevant facts. These facts are categorized into two types:
- Grounding Facts: These connect abstract concepts to specific terms found in the question and answer (e.g., linking "friction" to "rubbing together").
- Abstract Facts: These represent general scientific knowledge (e.g., "friction causes temperature to increase").
The system then uses Linear Programming (LP) to extract an optimal subgraph of these facts. This subgraph represents the most plausible explanation. The LP formulation is based on a "rooted maximum-weight connected subgraph problem" with constraints designed to emulate the grounding-abstract inference chains. The weights of nodes (facts) and edges (connections) in the graph are determined by three properties:
- Relevance: How well a fact relates to the hypothesis.
- Cohesion: How well different facts connect to maintain context.
- Diversity: How well facts cover different aspects of the hypothesis, avoiding redundancy.
These properties are combined using learnable weights that are optimized via Bayesian Optimization to achieve the best answer selection accuracy. This combination of LP for structured inference and Bayesian Optimization for parameter tuning is a novel aspect of their approach.
Datasets
The paper evaluates ExplanationLP on two established science question answering datasets:
- WorldTree Corpus: A dataset of 2,290 questions with explanation graphs, used for both training and evaluation.
- ARC-Challenge Corpus: A challenging multiple-choice question dataset from science exams (grades 3-9), used to test the model's robustness and performance on complex questions.
Key Insights and Ideas
- Explicitly Modeling Grounding and Abstraction: The paper highlights the importance of explicitly modeling both the concrete links (grounding) and the general scientific principles (abstraction) for effective science question answering.
- Linear Programming for Optimal Explanation Subgraphs: Using LP to select the best set of explanatory facts, rather than just a sequence, allows for a more structured and constrained inference process.
- Bayesian Optimization for Parameter Tuning: Combining LP with Bayesian Optimization provides an effective way to tune the complex weighting parameters that balance relevance, cohesion, and diversity.
- Robustness to Semantic Drift: The grounding-abstract chain mechanism appears to mitigate the "semantic drift" problem often seen in multi-hop reasoning, leading to more stable performance even with longer explanations.
Key Results
- Competitive Performance: ExplanationLP achieves comparable performance to transformer-based and multi-hop approaches on both WorldTree and ARC-Challenge datasets, despite having a significantly lower number of parameters (e.g., 9 parameters for ExplanationLP vs. 110M for BERT Base).
- Outperforming Baselines: On the WorldTree corpus, ExplanationLP outperforms BERT-based models by up to 11.66% in accuracy on the Easy split and 20.76% on the Challenge split. It also outperforms PathNet by significant margins.
- Explainability: The model generates plausible explanations, as demonstrated by case studies where the generated explanations align with gold explanations.
- Robustness: ExplanationLP shows better robustness to semantic drift compared to transformer-based and multi-hop approaches, maintaining performance with increasing explanation length.
- Ablation Study: The study confirms the importance of grounding-abstract chains, node/edge weights, and cohesion for the model's performance.
Limitations and Future Work
- Limitations: The paper acknowledges that while diversity plays a role, its impact is smaller compared to cohesion and relevance. The performance of the model is also dependent on the quality of the retrieved facts.
- Future Work: The authors suggest that further research could explore more sophisticated ways to integrate grounding and abstract mechanisms. The paper also implicitly points towards the potential for further optimization of the LP constraints and the Bayesian optimization process.
Summary updated