Abstract
This paper investigates the effective integration of external knowledge bases, such as ConceptNet and ATOMIC, into neural networks for commonsense question-answering (QA). We compare attention-based injection against knowledge pre-training, finding that the former is more surgical and effective. Our results demonstrate that the alignment between knowledge type and task domain is the primary driver of performance, with significant gains observed on the DREAM and CommonsenseQA benchmarks.
Introduction
While large pre-trained language models like BERT have mastered many standard QA benchmarks, they often stumble on "non-extractive" questions—those where the answer isn't explicitly written in the text but requires "reading between the lines." This is the realm of commonsense reasoning. This research seeks to determine which knowledge resources (relational vs. procedural) are most beneficial and identifies the optimal neural architecture to "inject" this knowledge without compromising the model's existing capabilities.
Methodology: The Option Comparison Network
The authors build their system on top of the Option Comparison Network (OCN). Unlike standard BERT-based QA, which evaluates answer choices in isolation, OCN explicitly models the relationships between different answer options, mimicking how humans weigh choices.
Knowledge Elicitation and Integration
To add "common sense," the study tested two main strategies across different knowledge bases:
- ConceptNet: A graph of general relations. Relevant triples were extracted to connect the question to the answers.
- ATOMIC: Focused on social "if-then" reasoning. A generative model (COMET) produced relevant social context (e.g., "If X does this, X wants to be seen as...").
Integration was evaluated using Attention-based Injection, which allows the model to "look up" relevant facts via a tri-linear attention mechanism, and Knowledge Pre-training, which fine-tunes the model on pseudo-sentences derived from the knowledge bases before task-specific training.
Experimental Results
The study evaluates these methods on two distinct benchmarks: DREAM (dialogue-based social reasoning) and CommonsenseQA (relational fact reasoning).
As shown in the results for CommonsenseQA, combining pre-training with attention-based injection yielded the highest overall performance boost of +4.9% over the OCN baseline. On the DREAM dataset, the OCN baseline (70.0% accuracy) outperformed standard BERT (66.8%), with ATOMIC injection providing the best results for questions involving social intentions.
Key Insights
- The "Domain Overlap" Rule: The type of knowledge must match the type of question. ATOMIC (social reasoning) helped significantly on DREAM, while ConceptNet (relational facts) was the clear winner for CommonsenseQA.
- Injection vs. Pre-training: Attention-based injection is generally more effective. Pre-training often shifted the model's weight distribution too much, leading to "catastrophic forgetting" of standard reasoning capabilities.
- The Negation Hurdle: Error analysis revealed that models still struggle with negation and antonyms. They can identify related concepts but often fail to reason that a concept is the opposite of what is required.
Conclusion and Future Work
The authors suggest that future systems should move toward conditional injection—only calling upon external knowledge when the model is uncertain. Furthermore, aggregating specialized resources into a Global Commonsense Knowledge Base will be essential for ensuring models have the right tools for diverse reasoning tasks.
Summary updated