Abstract
Automatic KB completion for commonsense knowledge graphs (e.g., ATOMIC and ConceptNet) poses unique challenges compared to the much studied conventional knowledge bases (e.g., Freebase). Commonsense knowledge graphs use free-form text to represent nodes, resulting in orders of magnitude more nodes compared to conventional KBs (18x more nodes in ATOMIC compared to Freebase (FB15K-237)). Importantly, this implies significantly sparser graph structures - a major challenge for existing KB completion methods that assume densely connected graphs over a relatively smaller set of nodes. In this paper, we present novel KB completion models that can address these challenges by exploiting the structural and semantic context of nodes. Specifically, we investigate two key ideas: (1) learning from local graph structure, using graph convolutional networks and automatic graph densification and (2) transfer learning from pre-trained language models to knowledge graphs for enhanced contextual representation of knowledge. We describe our method to incorporate information from both these sources in a joint model and provide the first empirical results for KB completion on ATOMIC and evaluation with ranking metrics on ConceptNet. Our results demonstrate the effectiveness of language model representations in boosting link prediction performance and the advantages of learning from local graph structure (+1.5 points in MRR for ConceptNet) when training on subgraphs for computational efficiency. Further analysis on model predictions shines light on the types of commonsense knowledge that language models capture well.
Introduction: The Challenge of Commonsense Knowledge
Traditional Knowledge Base (KB) completion methods were designed for "canonicalized" databases like Freebase, where entities are unique IDs and the graph is relatively dense. Commonsense KGs, such as ConceptNet and ATOMIC, present a different set of challenges. First, they utilize free-form text, where nodes are phrases (e.g., "prevent tooth decay") rather than single entities. This leads to a massive explosion in the number of nodes. Second, they exhibit extreme sparsity. Because nodes are not collapsed into single entities, the connections between them are incredibly sparse compared to traditional KBs.
Standard models like ConvE struggle in these sparse environments because they rely on dense local neighborhoods to learn representations. This research investigates bridging this gap by leveraging both structural (graph-based) and semantic (text-based) context.
Methodology: Integrating Structure and Semantics
The proposed joint model architecture integrates three main components to address the limitations of sparse graphs:
- Semantic Context (BERT): The model fine-tunes BERT on the KG's node phrases, capturing the inherent meaning of the text and performing transfer learning from language to knowledge.
- Structural Context (GCN) & Graph Densification: A Graph Convolutional Network (GCN) aggregates information from a node's local neighborhood. To overcome sparsity, the authors introduce "Graph Densification," adding synthetic edges between nodes with high semantic similarity based on BERT embeddings.
- Convolutional Decoder (ConvTransE): To score potential new triples, a convolutional approach is used that preserves the translational properties of the relationships.
To ensure the model learns from both sources, the authors utilize Progressive Masking, gradually unmasking BERT embeddings during training to prevent the model from over-relying on pre-trained semantic knowledge at the expense of graph structure.
Key Insights and Results
The models were evaluated on ConceptNet-100K and ATOMIC, using FB15K-237 as a scale reference.
- The Power of BERT: BERT provides a massive boost for ConceptNet, particularly for taxonomic relations like
IsAorMadeOf, which are frequently described in the text corpora BERT was trained on. The joint model achieved an MRR of 51.11 on ConceptNet, compared to the baseline score of 18.68. - Overcoming the Sparsity Wall: GCNs alone fail on commonsense KGs due to the lack of edges. Adding similarity-based synthetic edges proved crucial for making graph-based learning viable, particularly for ATOMIC, where the best model achieved an MRR of 14.44.
- Social Logic vs. Taxonomy: BERT was found to be less effective for ATOMIC than for ConceptNet. ATOMIC requires understanding social dynamics and intent, which are less explicitly mapped in standard text than simple factual taxonomies.
- Human Evaluation: Because these KGs are sparse, many valid predicted facts are marked as "false negatives" because they aren't in the test set. Human judges found that ~74% of the top-10 predictions for both datasets were actually valid.
Conclusion
This work demonstrates that combining structural graph information with semantic language representations is essential for completing large-scale, sparse commonsense knowledge bases. While computational intensity remains a challenge due to the massive graph sizes, the integration of synthetic densification and pre-trained language models offers a robust path forward for enhancing machine reasoning with commonsense knowledge.
Summary updated