Abstract
Non-extractive commonsense QA remains a challenging AI task, as it requires systems to reason about, synthesize, and gather disparate pieces of information, in order to generate responses to queries. Recent approaches on such tasks show increased performance, only when models are either pre-trained with additional information or when domain-specific heuristics are used, without any special consideration regarding the knowledge resource type. In this paper, we perform a survey of recent commonsense QA methods and we provide a systematic analysis of popular knowledge resources and knowledge-integration methods, across benchmarks from multiple commonsense datasets. Our results and analysis show that attention-based injection seems to be a preferable choice for knowledge integration and that the degree of domain overlap, between knowledge bases and datasets, plays a crucial role in determining model success.
Introduction
While large pre-trained language models like BERT have mastered extractive question answering—finding an answer explicitly stated in a text—they often struggle with non-extractive commonsense QA. These tasks require "reading between the lines" using general world knowledge not explicitly provided in the prompt. For instance, understanding that "rushing to the hospital" implies a person is likely "worried" or "injured" requires social and procedural knowledge beyond simple text matching.
The challenge in building effective neuro-symbolic systems is twofold: first, the diversity of commonsense knowledge (factual, taxonomic, procedural, social); and second, the mechanics of integration. Researchers must decide whether knowledge should be "pre-trained" into the model weights or "injected" dynamically during inference using attention mechanisms.
Methodology
The proposed system is built upon the Option Comparison Network (OCN). Unlike standard models that encode each answer choice independently, OCN explicitly models pairwise interactions between options, mimicking human decision-making processes when weighing choices.
To enhance this architecture, the authors integrate two primary knowledge sources:
- ConceptNet: A semantic network for general taxonomic and relational knowledge (e.g., "a cat is a mammal").
- ATOMIC: A knowledge base focused on "if-then" social commonsense (e.g., intentions and reactions).
Two integration strategies are compared:
- Attention-Based Injection: A BiLSTM encodes knowledge triples, and a tri-linear attention mechanism fuses this information into the model's hidden states at inference time.
- Knowledge Pre-training: BERT is fine-tuned on "pseudo-sentences" derived from knowledge triples before training on the specific QA task.
Experimental Setup
The study evaluates these methods on two distinct benchmarks:
- DREAM: A dialogue-based multiple-choice QA dataset requiring understanding of social implications and speaker intentions.
- CommonsenseQA: A dataset designed to test ConceptNet-style reasoning, where questions are derived from ConceptNet subgraphs.
Results and Analysis
The most critical finding is the Alignment Rule: the effectiveness of a knowledge base depends heavily on its overlap with the dataset's domain. ConceptNet injection provided a significant boost for CommonsenseQA but was less effective for the social-dialogue-heavy DREAM dataset, where ATOMIC proved more useful.
Furthermore, Injection is more robust than Pre-training. Pre-training on specific knowledge bases can "warp" model weights, leading to performance drops on general questions. In contrast, attention-based injection is more "surgical," allowing the model to selectively use external knowledge only when relevant.
On CommonsenseQA, the OCN baseline (64.1% Dev Acc) improved to 67.3% with ConceptNet injection. Combining OMCS pre-training with injection reached 69.0%, demonstrating complementarity when domain alignment is high. Conversely, on DREAM, ATOMIC injection improved the baseline to 70.1% Test Acc, while pre-training caused a significant performance drop to ~60%, highlighting the risks of over-tuning.
Conclusion
While knowledge injection provides clear benefits, error analysis reveals that systems still struggle with logical negations and "Antonym" questions. Future work aims to automate the selection of knowledge bases and develop more robust "conditional" injection methods to prevent external knowledge from becoming a distractor when it isn't needed.
Summary updated