Towards Generalizable Neuro-Symbolic Systems for Commonsense Question Answering

Kaixin Ma, Jonathan Francis, Quanyang Lu, Eric Nyberg, Alessandro Oltramari

Abstract

Non-extractive commonsense QA remains a challenging AI task, as it requires systems to reason about, synthesize, and gather disparate pieces of information, in order to generate responses to queries. Recent approaches on such tasks show increased performance, only when models are either pre-trained with additional information or when domain-specific heuristics are used, without any special consideration regarding the knowledge resource type. In this paper, we perform a survey of recent commonsense QA methods and we provide a systematic analysis of popular knowledge resources and knowledge-integration methods, across benchmarks from multiple commonsense datasets. Our results and analysis show that attention-based injection seems to be a preferable choice for knowledge integration and that the degree of domain overlap, between knowledge bases and datasets, plays a crucial role in determining model success.

Introduction

While large pre-trained language models like BERT have mastered extractive question answering—finding an answer explicitly stated in a text—they often struggle with non-extractive commonsense QA. These tasks require "reading between the lines" using general world knowledge not explicitly provided in the prompt. For instance, understanding that "rushing to the hospital" implies a person is likely "worried" or "injured" requires social and procedural knowledge beyond simple text matching.

The challenge in building effective neuro-symbolic systems is twofold: first, the diversity of commonsense knowledge (factual, taxonomic, procedural, social); and second, the mechanics of integration. Researchers must decide whether knowledge should be "pre-trained" into the model weights or "injected" dynamically during inference using attention mechanisms.

Methodology

The proposed system is built upon the Option Comparison Network (OCN). Unlike standard models that encode each answer choice independently, OCN explicitly models pairwise interactions between options, mimicking human decision-making processes when weighing choices.

To enhance this architecture, the authors integrate two primary knowledge sources:

  • ConceptNet: A semantic network for general taxonomic and relational knowledge (e.g., "a cat is a mammal").
  • ATOMIC: A knowledge base focused on "if-then" social commonsense (e.g., intentions and reactions).

Two integration strategies are compared:

  1. Attention-Based Injection: A BiLSTM encodes knowledge triples, and a tri-linear attention mechanism fuses this information into the model's hidden states at inference time.
  2. Knowledge Pre-training: BERT is fine-tuned on "pseudo-sentences" derived from knowledge triples before training on the specific QA task.
The overall architecture of the proposed system, featuring the Inject Cell for knowledge integration and the Option Comparison Cell.

Experimental Setup

The study evaluates these methods on two distinct benchmarks:

  • DREAM: A dialogue-based multiple-choice QA dataset requiring understanding of social implications and speaker intentions.
  • CommonsenseQA: A dataset designed to test ConceptNet-style reasoning, where questions are derived from ConceptNet subgraphs.

Results and Analysis

The most critical finding is the Alignment Rule: the effectiveness of a knowledge base depends heavily on its overlap with the dataset's domain. ConceptNet injection provided a significant boost for CommonsenseQA but was less effective for the social-dialogue-heavy DREAM dataset, where ATOMIC proved more useful.

Furthermore, Injection is more robust than Pre-training. Pre-training on specific knowledge bases can "warp" model weights, leading to performance drops on general questions. In contrast, attention-based injection is more "surgical," allowing the model to selectively use external knowledge only when relevant.

Performance comparison on the CommonsenseQA dataset, showing the impact of ConceptNet (CN) injection and pre-training.

On CommonsenseQA, the OCN baseline (64.1% Dev Acc) improved to 67.3% with ConceptNet injection. Combining OMCS pre-training with injection reached 69.0%, demonstrating complementarity when domain alignment is high. Conversely, on DREAM, ATOMIC injection improved the baseline to 70.1% Test Acc, while pre-training caused a significant performance drop to ~60%, highlighting the risks of over-tuning.

Conclusion

While knowledge injection provides clear benefits, error analysis reveals that systems still struggle with logical negations and "Antonym" questions. Future work aims to automate the selection of knowledge bases and develop more robust "conditional" injection methods to prevent external knowledge from becoming a distractor when it isn't needed.

Summary updated

Abstract

Non-extractive commonsense QA remains a challenging AI task, as it requires systems to reason about, synthesize, and gather disparate pieces of information, in order to generate responses to queries. Recent approaches on such tasks show increased performance, only when models are either pre-trained with additional information or when domain-specific heuristics are used, without any special consideration regarding the knowledge resource type. In this paper, we perform a survey of recent commonsense QA methods and we provide a systematic analysis of popular knowledge resources and knowledge-integration methods, across benchmarks from multiple commonsense datasets. Our results and analysis show that attention-based injection seems to be a preferable choice for knowledge integration and that the degree of domain overlap, between knowledge bases and datasets, plays a crucial role in determining model success.
BibTeX
@misc{ma2019towards,
  author = {Kaixin Ma and Jonathan Francis and Quanyang Lu and Eric Nyberg and Alessandro Oltramari},
  title = {{Towards Generalizable Neuro-Symbolic Systems for Commonsense Question Answering}},
  year = {2019},
  eprint = {1910.14087},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1910.14087},
}
Figure