Abstract
Automatically classifying commonsense relations in large-scale knowledge graphs like ConceptNet presents unique challenges due to multi-word expressions, overlapping labels, and inherent database incompleteness. This paper explores these difficulties by treating relation classification as a multi-label problem. We propose a neural architecture that leverages Bi-LSTMs for complex argument encoding and employs relation-specific probability thresholds. Our results indicate that while certain relations are highly predictable, others suffer from significant semantic ambiguity. We demonstrate that customized thresholding and sequence-aware encoding are critical for improving classification performance in both closed and open-world scenarios.
Introduction
Large-scale knowledge bases (KBs) such as ConceptNet are foundational for Natural Language Understanding (NLU), yet their "messy" real-world nature often complicates automated reasoning. Traditional approaches frequently simplify relation classification into a one-to-one mapping. However, this study identifies three primary hurdles:
- Multi-labeling: Concept pairs often share multiple valid relations (e.g., "people" and "eating in groups" can be both DESIRES and CAPABLEOF).
- Argument Complexity: Concepts are frequently multi-word expressions (MWEs) with varying phrase types (Noun Phrases, Verb Phrases, etc.).
- Incompleteness: KBs are inherently incomplete, leading to "false positives" that may actually be valid but missing relations.
Methodology
To address these challenges, we developed a neural multi-label classifier. The architecture, illustrated in Figure 1, processes concept pairs through several stages:
- Input Representation: We compared centroid vectors (averaging embeddings) against Bi-LSTMs (RNNs) to capture the sequential nature of multi-word concepts.
- Architecture: Concept embeddings ($v_{e1}, v_{e2}$) pass through a tuning layer and are combined via operations like subtraction or concatenation before entering a Feed-Forward Neural Network (FFNN).
- Multi-Label Output: A sigmoid layer replaces the standard softmax to allow for independent prediction of multiple relations.
- Relation-Specific Thresholding: Instead of a global 0.5 cutoff, we tuned individual thresholds for each relation to optimize the balance between precision and recall.
Experimental Setup
The study utilizes the English Open Mind Common Sense (OMCS) portion of ConceptNet 5.6. We evaluated models on:
- CN-OMCS-14: A balanced subset of the 14 most frequent relations.
- Closed vs. Open World: Testing included "Closed World" (known relations only) and "Open World" (OW) settings, which introduce RANDOM pairs and OTHER relations to simulate real-world uncertainty.
Results and Discussion
Our analysis revealed several key insights into the mechanics of commonsense classification:
- The Impact of RNNs and Thresholding: As shown in Figure 2, using RNNs to encode phrases provided a significant performance boost over centroid averaging, particularly for relations involving complex verb phrases. Furthermore, individual threshold tuning was essential for harder-to-classify relations.
- Ambiguity and Overlap: We found a strong inverse correlation (-0.82) between performance and label overlap. Relations like HasSubevent and HasPrerequisite frequently co-occur, making them difficult to distinguish.
- Evaluation Bias: A manual annotation study revealed that approximately 9% of the model's "false positives" were actually correct relations missing from ConceptNet, suggesting that standard metrics may underestimate model utility.
Conclusion
This work highlights that the primary difficulty in ConceptNet relation classification stems from semantic ambiguity rather than just model capacity. While RNN-based encoding and customized thresholding significantly improve results, resolving overlapping labels remains a challenge. Future research should focus on meta-classification strategies to model relation co-occurrence and expanding evaluation frameworks to account for knowledge base incompleteness.
Summary updated