Introduction
Knowledge Graphs (KGs) such as Wikidata are typically structured into two distinct but interconnected layers: an entity graph representing factual instances and a concept graph representing higher-level ontologies. For example, while the entity graph might link Da Vinci to the Mona Lisa via a paint relation, the concept graph defines the relationship between painter and painting.
Despite the richness of these dual-view structures, most existing KG benchmarks focus almost exclusively on the entity graph. This focus limits the ability to evaluate models on human-like cognitive tasks such as abstraction (linking entities to concepts) and concretization (finding entities for a concept). The KACC benchmark introduces a unified framework to address these gaps.
The KACC Benchmark and Tasks
KACC evaluates models across three primary task categories:
- Knowledge Abstraction (KA & MKA): Predicting the tail for
instanceOfandsubclassOfrelations. Multi-hop KA (MKA) specifically tests the ability to infer upper-level concepts multiple steps away in the hierarchy. - Knowledge Concretization (KC & MKC): The inverse task of predicting specific entities or concepts given a higher-level category.
- Knowledge Completion: Standard link prediction for logical relations, evaluated in both "Single" (isolated graph) and "Joint" (cross-view) settings.
Datasets
The authors developed three new datasets—KACC-S, KACC-M, and KACC-L—sampled from Wikidata using FB15K-237 entities as seeds. As shown in the table below, these datasets provide a significant increase in the scale of concept graphs and the density of cross-links compared to previous benchmarks like YAGO26K and DB111K.
Key Insights and Results
Experiments using standard Knowledge Embedding (KE) models and specialized EC-KG models (such as AttH and JOIE) yielded several critical observations:
- Geometry and Hierarchy: Models utilizing hyperbolic geometry (e.g., AttH) significantly outperform Euclidean models on hierarchical tasks like abstraction. This is attributed to hyperbolic space's natural ability to represent tree-like structures.
- Directional Difficulty: Concretization is consistently more challenging than abstraction. Models find it easier to categorize a specific entity than to retrieve all correct entities for a broad concept.
- The Multi-hop Reasoning Gap: All models exhibit a "performance cliff" when moving from single-hop to multi-hop tasks. For instance, AttH's MRR dropped from 0.778 to 0.218 on the
instanceOftask when multi-hop reasoning was required. - Synergy of Joint Modeling: Models generally perform better in the "Joint" setting, indicating that the concept graph provides essential ontological constraints that aid in factual link prediction.
Conclusion
The KACC benchmark reveals that while current models are proficient at simple link prediction, they struggle with the transitive reasoning required for deep hierarchical understanding. Future work should explore contextualized embeddings and the integration of natural language to bridge the gap between symbolic knowledge and commonsense reasoning.
Summary updated