Abstract
Commonsense knowledge graphs (KGs) like ConceptNet and ATOMIC are significantly larger and sparser than traditional encyclopedic KBs, posing challenges for standard completion models. We propose a hybrid model that integrates structural context from Graph Convolutional Networks (GCNs) with semantic insights from pre-trained language models (BERT). To address extreme sparsity, we introduce a "graph densification" technique using synthetic similarity-based edges. Our approach demonstrates significant improvements in MRR and human-validated accuracy across major commonsense benchmarks.
Introduction: The Sparsity Challenge in Commonsense KGs
Traditional Knowledge Base (KB) completion research typically focuses on "encyclopedic" datasets like Freebase. However, commonsense knowledge graphs are fundamentally different. While a traditional KB might have a few thousand canonical entities, a commonsense KG uses free-form text (e.g., "prevent tooth decay" vs. "tooth decay"), leading to a massive explosion in the number of nodes. For instance, ATOMIC contains roughly 18 times more nodes than Freebase.
Because these nodes are so diverse, the resulting graphs are incredibly sparse; ConceptNet is approximately 100 times less dense than standard benchmarks. Existing completion models, which rely on dense connectivity to propagate information, struggle in these "knowledge deserts" where many nodes exist as isolated islands.
Methodology: Integrating Structure and Semantics
To overcome these challenges, we introduce a model that integrates three main components:
- Semantic Context (BERT): We fine-tune BERT on the natural language phrases within the KG nodes. This allows the model to leverage the vast "latent" commonsense knowledge BERT acquired during its original pre-training.
- Structural Context (GCN): A Graph Convolutional Network encodes nodes based on their neighbors. To make this effective on sparse graphs, we implement Graph Densification. By calculating semantic similarity between all nodes and adding "synthetic" edges between highly similar ones, we provide the GCN with more paths for information propagation.
- Progressive Masking for Fusion: To prevent the model from over-relying on BERT and ignoring the graph structure, we employ a "progressive masking" strategy. At the start of training, BERT embeddings are heavily masked, forcing the model to learn from the graph. This mask is gradually lifted over 100 epochs.
The final scores are calculated using a ConvTransE decoder, which applies convolutional filters to the combined entity and relation embeddings.
Datasets and Evaluation
The study evaluates performance on two primary commonsense benchmarks:
- ConceptNet-100K: General taxonomic and functional knowledge (e.g.,
cake—MadeOf—sugar). - ATOMIC: Social commonsense, including causes, effects, and intents (e.g.,
PersonX pays PersonY—xIntent—to be fair). - FB15K-237: A reference point illustrating the density gap between traditional and commonsense KBs.
Key Insights
- Domain-Specific Strengths: BERT provides a massive boost (up to 35 points in MRR) for ConceptNet, likely because taxonomic relations (like
IsA) are frequently described in its training corpora. However, social "if-then" reasoning in ATOMIC is less explicitly stated in general text, making the graph's structural context more vital. - The Necessity of Densification: Standard GCNs fail on sparse graphs. By adding synthetic similarity-based edges, we effectively "irrigate" the graph, allowing information to flow between conceptually related but technically disconnected nodes.
- The "False Negative" Problem: Due to sparsity, models often predict valid facts not present in the test set. Human evaluation reveals that models perform better than automatic metrics like MRR suggest.
Results
On ConceptNet-100K, the full model (SIM + GCN + BERT + ConvTransE) achieved an MRR of 51.11, a significant improvement over the standard ConvTransE baseline of 18.68. On ATOMIC, the structural GCN components were more critical; adding similarity-induced edges provided a +1.5 MRR boost over models using only BERT and standard relations.
In blind human evaluation tests, the top-10 predictions of the best model were rated as "valid" 74.16% of the time for ConceptNet and 73.96% for ATOMIC, significantly outperforming models that lacked graph densification.
Limitations and Future Directions
Training on massive, densified graphs is computationally intensive; the ATOMIC model required approximately 30GB of VRAM and 8-10 days of training. Future work will explore more efficient subgraph sampling and learnable densification thresholds. Additionally, while our discriminative approach excels at ranking, merging it with generative paradigms (like COMET) could offer greater flexibility in creating entirely new commonsense phrases.
Summary updated