Commonsense Knowledge Base Completion with Structural and Semantic Context

Chaitanya Malaviya, Chandra Bhagavatula, Antoine Bosselut, Yejin Choi

Abstract

Commonsense knowledge graphs (KGs) like ConceptNet and ATOMIC are significantly larger and sparser than traditional encyclopedic KBs, posing challenges for standard completion models. We propose a hybrid model that integrates structural context from Graph Convolutional Networks (GCNs) with semantic insights from pre-trained language models (BERT). To address extreme sparsity, we introduce a "graph densification" technique using synthetic similarity-based edges. Our approach demonstrates significant improvements in MRR and human-validated accuracy across major commonsense benchmarks.

Introduction: The Sparsity Challenge in Commonsense KGs

Traditional Knowledge Base (KB) completion research typically focuses on "encyclopedic" datasets like Freebase. However, commonsense knowledge graphs are fundamentally different. While a traditional KB might have a few thousand canonical entities, a commonsense KG uses free-form text (e.g., "prevent tooth decay" vs. "tooth decay"), leading to a massive explosion in the number of nodes. For instance, ATOMIC contains roughly 18 times more nodes than Freebase.

Because these nodes are so diverse, the resulting graphs are incredibly sparse; ConceptNet is approximately 100 times less dense than standard benchmarks. Existing completion models, which rely on dense connectivity to propagate information, struggle in these "knowledge deserts" where many nodes exist as isolated islands.

Methodology: Integrating Structure and Semantics

To overcome these challenges, we introduce a model that integrates three main components:

  1. Semantic Context (BERT): We fine-tune BERT on the natural language phrases within the KG nodes. This allows the model to leverage the vast "latent" commonsense knowledge BERT acquired during its original pre-training.
  2. Structural Context (GCN): A Graph Convolutional Network encodes nodes based on their neighbors. To make this effective on sparse graphs, we implement Graph Densification. By calculating semantic similarity between all nodes and adding "synthetic" edges between highly similar ones, we provide the GCN with more paths for information propagation.
  3. Progressive Masking for Fusion: To prevent the model from over-relying on BERT and ignoring the graph structure, we employ a "progressive masking" strategy. At the start of training, BERT embeddings are heavily masked, forcing the model to learn from the graph. This mask is gradually lifted over 100 epochs.

The final scores are calculated using a ConvTransE decoder, which applies convolutional filters to the combined entity and relation embeddings.

The proposed model architecture featuring an encoder with GCN and BERT components and a ConvTransE decoder.

Datasets and Evaluation

The study evaluates performance on two primary commonsense benchmarks:

  • ConceptNet-100K: General taxonomic and functional knowledge (e.g., cake — MadeOf — sugar).
  • ATOMIC: Social commonsense, including causes, effects, and intents (e.g., PersonX pays PersonY — xIntent — to be fair).
  • FB15K-237: A reference point illustrating the density gap between traditional and commonsense KBs.

Key Insights

  • Domain-Specific Strengths: BERT provides a massive boost (up to 35 points in MRR) for ConceptNet, likely because taxonomic relations (like IsA) are frequently described in its training corpora. However, social "if-then" reasoning in ATOMIC is less explicitly stated in general text, making the graph's structural context more vital.
  • The Necessity of Densification: Standard GCNs fail on sparse graphs. By adding synthetic similarity-based edges, we effectively "irrigate" the graph, allowing information to flow between conceptually related but technically disconnected nodes.
  • The "False Negative" Problem: Due to sparsity, models often predict valid facts not present in the test set. Human evaluation reveals that models perform better than automatic metrics like MRR suggest.

Results

On ConceptNet-100K, the full model (SIM + GCN + BERT + ConvTransE) achieved an MRR of 51.11, a significant improvement over the standard ConvTransE baseline of 18.68. On ATOMIC, the structural GCN components were more critical; adding similarity-induced edges provided a +1.5 MRR boost over models using only BERT and standard relations.

Human evaluation results comparing different model configurations on CN-100K and ATOMIC datasets.

In blind human evaluation tests, the top-10 predictions of the best model were rated as "valid" 74.16% of the time for ConceptNet and 73.96% for ATOMIC, significantly outperforming models that lacked graph densification.

Limitations and Future Directions

Training on massive, densified graphs is computationally intensive; the ATOMIC model required approximately 30GB of VRAM and 8-10 days of training. Future work will explore more efficient subgraph sampling and learnable densification thresholds. Additionally, while our discriminative approach excels at ranking, merging it with generative paradigms (like COMET) could offer greater flexibility in creating entirely new commonsense phrases.

Summary updated

Abstract

Automatic KB completion for commonsense knowledge graphs (e.g., ATOMIC and ConceptNet) poses unique challenges compared to the much studied conventional knowledge bases (e.g., Freebase). Commonsense knowledge graphs use free-form text to represent nodes, resulting in orders of magnitude more nodes compared to conventional KBs ( ∼18x more nodes in ATOMIC compared to Freebase (FB15K-237)). Importantly, this implies significantly sparser graph structures — a major challenge for existing KB completion methods that assume densely connected graphs over a relatively smaller set of nodes.In this paper, we present novel KB completion models that can address these challenges by exploiting the structural and semantic context of nodes. Specifically, we investigate two key ideas: (1) learning from local graph structure, using graph convolutional networks and automatic graph densification and (2) transfer learning from pre-trained language models to knowledge graphs for enhanced contextual representation of knowledge. We describe our method to incorporate information from both these sources in a joint model and provide the first empirical results for KB completion on ATOMIC and evaluation with ranking metrics on ConceptNet. Our results demonstrate the effectiveness of language model representations in boosting link prediction performance and the advantages of learning from local graph structure (+1.5 points in MRR for ConceptNet) when training on subgraphs for computational efficiency. Further analysis on model predictions shines light on the types of commonsense knowledge that language models capture well.
BibTeX
@inproceedings{malaviya2020commonsense,
  author = {Chaitanya Malaviya and Chandra Bhagavatula and Antoine Bosselut and Yejin Choi},
  title = {{Commonsense Knowledge Base Completion with Structural and Semantic Context}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2020},
  doi = {10.1609/aaai.v34i03.5684},
}
Figure