Answering Elementary Science Questions by Constructing Coherent Scenes using Background Knowledge

Yang Li, Peter ClarkEMNLP 2015 · OpenReview

Abstract

Much of what we understand from text is not explicitly stated. Rather, the reader uses his/her knowledge to fill in gaps and create a coherent, mental picture or “scene” depicting what text appears to convey. The scene constitutes an understanding of the text, and can be used to answer questions that go beyond the text. Our goal is to answer elementary science questions, where this requirement is pervasive; A question will often give a partial description of a scene and ask the student about implicit information. We show that by using a simple “knowledge graph” representation of the question, we can leverage several large-scale linguistic resources to provide missing background knowledge, somewhat alleviating the knowledge bottleneck in previous approaches. The coherence of the best resulting scene, built from a question/answer-candidate pair, reflects the confidence that the answer candidate is correct, and thus can be used to answer multiple choice questions. Our experiments show that this approach outperforms competitive algorithms on several datasets tested. The significance of this work is thus to show that a simple “knowledge graph” representation allows a version of “interpretation as scene construction” to be made viable.

Introduction

Natural Language Understanding (NLU) often requires bridging the gap between explicit text and the implicit "common sense" knowledge needed to interpret it. While factoid questions can often be solved through keyword matching, elementary science questions are deceptively complex. For example, answering why a baby's rattle makes noise requires understanding that shaking involves movement, which is a form of mechanical energy. This paper introduces SceneQA, a system that addresses this "knowledge bottleneck" by constructing mental models, or "scenes," represented as knowledge graphs.

The SceneQA Methodology

The authors propose an "elaborate-and-prune" algorithm to simulate the human process of scene construction. This process transforms a question into a dense, coherent graph that connects the query to potential answers.

The four-step SceneQA pipeline: from question analysis to scoring the best scene.

1. Question Analysis

The system first extracts keywords from the question and assigns importance scores based on their frequency in Google search snippets, which serves as a proxy for relevance in a scientific context.

2. Knowledge Elaboration (The Builder)

To fill in missing information, the system fetches "triples" (subject-relation-object) from three massive linguistic resources. These triples introduce "neighbor" nodes to the graph, providing the connective tissue between disparate concepts (e.g., connecting "pedaling" to "energy" via the intermediate node "power").

Knowledge bases used for scene construction and their respective sizes.

3. Scene Pruning

For each multiple-choice answer candidate, the system adds that candidate to the graph. It then iteratively prunes the graph, removing nodes that are weakly connected. The goal is to find the most coherent subgraph that links the question keywords to the specific answer candidate.

4. Scoring

The system calculates a "coherence score" for the final scene. The system uses word2vec cosine similarity to weight the strength of relationships between nodes. The answer candidate that produces the most well-connected and dense graph is selected as the winner.

Experimental Evaluation

SceneQA was evaluated on non-diagram, multiple-choice questions from real-world NY Regents science exams (4th and 5th grade). The system was compared against two competitive baselines: LSModel (a lexical semantics model) and A*Rules (an inference-rule-based system).

Results showed that SceneQA significantly outperformed the baselines across all test sets. On the largest dataset (Test3), SceneQA achieved 55.20% accuracy, compared to roughly 30% for the baselines. On Test2, the gap was even wider, with SceneQA reaching 65.38% accuracy. Ablation studies confirmed that both the elaboration (adding knowledge) and pruning (removing noise) steps are essential for performance.

Limitations and Future Work

Despite its success, the model has limitations. It currently treats questions as a "bag of words," ignoring syntactic structure and logical operators like negation (e.g., "Which of these is not..."). It also struggles with questions involving sequences or processes. Future work will focus on integrating richer linguistic structures and better methods for contextualizing relationship strengths within the knowledge graphs.

Conclusion

SceneQA demonstrates that viewing NLU as a process of scene construction is a viable and effective approach for complex question-answering. By leveraging external knowledge to fill in the blanks, machines can move beyond simple pattern matching toward a more human-like understanding of scientific scenarios.

Summary updated

Abstract

Much of what we understand from text is not explicitly stated. Rather, the reader uses his/her knowledge to fill in gaps and create a coherent, mental picture or “scene” depicting what text appears to convey. The scene constitutes an understanding of the text, and can be used to answer questions that go beyond the text. Our goal is to answer elementary science questions, where this requirement is pervasive; A question will often give a partial description of a scene and ask the student about implicit information. We show that by using a simple “knowledge graph” representation of the question, we can leverage several large-scale linguistic resources to provide missing background knowledge, somewhat alleviating the knowledge bottleneck in previous approaches. The coherence of the best resulting scene, built from a question/answer-candidate pair, reflects the confidence that the answer candidate is correct, and thus can be used to answer multiple choice questions. Our experiments show that this approach outperforms competitive algorithms on several datasets tested. The significance of this work is thus to show that a simple “knowledge graph” representation allows a version of “interpretation as scene construction” to be made viable.
BibTeX
@inproceedings{li2015answering,
  author = {Yang Li and Peter Clark},
  title = {{Answering Elementary Science Questions by Constructing Coherent Scenes using Background Knowledge}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2015},
  url = {https://openreview.net/forum?id=BybELGMOWB},
}
Figure