Coarse-To-Careful: Seeking Semantic-Related Knowledge for Open-Domain Commonsense Question Answering

Luxi Xing, Yue Hu, Jing Yu, Yuqiang Xie, Wei Peng

Introduction

Pre-trained Language Models (PLMs) such as BERT and RoBERTa have revolutionized natural language processing through their ability to capture complex text patterns. However, these models often falter when faced with questions requiring implicit, real-world commonsense knowledge not explicitly present in the training corpus. While integrating external structured knowledge graphs like ConceptNet is a common solution, it introduces a significant challenge: noise. Blindly injecting vast amounts of external knowledge can lead to the inclusion of misleading or irrelevant information, potentially degrading model performance.

The SEEK-QA Framework

To address the noise issue, this paper proposes SEEK-QA (Semantic-driven Knowledge-aware Question Answering), a framework designed to selectively filter and inject external knowledge using a "coarse-to-careful" approach.

The SEEK-QA framework architecture, featuring the SONAR filtering stage and the SKETCH fusion module.

The framework operates via two primary novel components:

1. SONAR: Semantic Monitoring Knowledge Tailoring

SONAR acts as the "coarse" filtering stage. It extracts potential link paths between concepts in the question and candidate answers from ConceptNet. These paths are then scored based on link validity, concept similarity, and relation similarity using GloVe embeddings. By applying heuristic thresholds, SONAR discards irrelevant paths, effectively denoising the input before it reaches the reasoning stage.

2. SKETCH: Semantic-aware Knowledge Fetching

SKETCH serves as the "careful" fusion module. It utilizes a Relation-aware Graph Attention Network (GAT) and BiGRUs to encode the filtered knowledge graph. SKETCH calculates two hierarchical attention weights:

  • Semantic Link Strength (SLS): Measures the relevance of a specific knowledge path to a concept pair.
  • Semantic Union Strength (SUS): Measures the relevance of that concept pair to the overall question context.

This hierarchical weighting allows the model to selectively merge the most critical external knowledge with the PLM's contextual encoding via a selective gate.

Experimental Results

The SEEK-QA framework was evaluated on the CommonsenseQA benchmark, using ConceptNet as the external knowledge base.

Accuracy comparison of SEEK-QA against various baseline models on the CommonsenseQA development set.

As shown in the results, SEEK-QA achieved an accuracy of 74.52%, significantly outperforming standard PLMs like RoBERTa-large (71.44%) and previous knowledge-aware architectures such as KagNet (64.46%).

Ablation studies further confirmed the importance of each component. Removing the SONAR filtering step dropped performance to 73.62%, while removing the SLS or SUS components from SKETCH led to drops of approximately 2-3%, validating the necessity of both coarse denoising and careful hierarchical fusion.

Key Insights

  • Selective Injection is Crucial: Simply increasing the volume of external knowledge is counterproductive. Aggressive pruning of irrelevant triples through semantic similarity (SONAR) makes the model more focused and efficient.
  • Hierarchical Relevance: Not all paths between concepts are equally informative. By distinguishing between path-level relevance (link strength) and concept-level relevance to the question (union strength), SKETCH provides a more nuanced grounding mechanism.
  • Augmenting PLMs: While PLMs possess significant internal knowledge, explicit structured knowledge—when carefully filtered—remains a powerful tool for complex reasoning tasks.

Limitations and Future Work

The current framework relies on manually tuned heuristic thresholds for filtering, which may require adjustment for different datasets. Future research could explore automated methods for learning these thresholds, extending the framework to generative QA tasks, or integrating multiple heterogeneous knowledge sources simultaneously.

Summary updated

Abstract

It is prevalent to utilize external knowledge to help machine answer questions that need background commonsense, which faces a problem that unlimited knowledge will transmit noisy and misleading information. Towards the issue of introducing related knowledge, we propose a semantic-driven knowledge-aware QA framework, which controls the knowledge injection in a coarse-to-careful fashion. We devise a tailoring strategy to filter extracted knowledge under monitoring of the coarse semantic of question on the knowledge extraction stage. And we develop a semantic-aware knowledge fetching module that engages structural knowledge information and fuses proper knowledge according to the careful semantic of questions in a hierarchical way. Experiments demonstrate that the proposed approach promotes the performance on the CommonsenseQA dataset comparing with strong baselines.
BibTeX
@inproceedings{xing2021coarse,
  author = {Luxi Xing and Yue Hu and Jing Yu and Yuqiang Xie and Wei Peng},
  title = {{Coarse-To-Careful: Seeking Semantic-Related Knowledge for Open-Domain Commonsense Question Answering}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2021},
  url = {https://openreview.net/forum?id=fAk8P5_CIE7},
}
Figure