KACC: A Multi-task Benchmark for Knowledge Abstraction, Concretization and Completion

Jie Zhou, Xin Lv, Cheng Yang, Zhiyuan Liu, Juanzi Li, Maosong Sun

Introduction

Knowledge Graphs (KGs) such as Wikidata are typically structured into two distinct but interconnected layers: an entity graph representing factual instances and a concept graph representing higher-level ontologies. For example, while the entity graph might link Da Vinci to the Mona Lisa via a paint relation, the concept graph defines the relationship between painter and painting.

Overview of an Entity-Concept Knowledge Graph (EC-KG) showing the hierarchical and logical relations between entities and concepts.

Despite the richness of these dual-view structures, most existing KG benchmarks focus almost exclusively on the entity graph. This focus limits the ability to evaluate models on human-like cognitive tasks such as abstraction (linking entities to concepts) and concretization (finding entities for a concept). The KACC benchmark introduces a unified framework to address these gaps.

The KACC Benchmark and Tasks

KACC evaluates models across three primary task categories:

  • Knowledge Abstraction (KA & MKA): Predicting the tail for instanceOf and subclassOf relations. Multi-hop KA (MKA) specifically tests the ability to infer upper-level concepts multiple steps away in the hierarchy.
  • Knowledge Concretization (KC & MKC): The inverse task of predicting specific entities or concepts given a higher-level category.
  • Knowledge Completion: Standard link prediction for logical relations, evaluated in both "Single" (isolated graph) and "Joint" (cross-view) settings.

Datasets

The authors developed three new datasets—KACC-S, KACC-M, and KACC-L—sampled from Wikidata using FB15K-237 entities as seeds. As shown in the table below, these datasets provide a significant increase in the scale of concept graphs and the density of cross-links compared to previous benchmarks like YAGO26K and DB111K.

Statistical comparison of KACC datasets (S, M, L) against existing benchmarks like YAGO26K and DB111K.

Key Insights and Results

Experiments using standard Knowledge Embedding (KE) models and specialized EC-KG models (such as AttH and JOIE) yielded several critical observations:

  1. Geometry and Hierarchy: Models utilizing hyperbolic geometry (e.g., AttH) significantly outperform Euclidean models on hierarchical tasks like abstraction. This is attributed to hyperbolic space's natural ability to represent tree-like structures.
  2. Directional Difficulty: Concretization is consistently more challenging than abstraction. Models find it easier to categorize a specific entity than to retrieve all correct entities for a broad concept.
  3. The Multi-hop Reasoning Gap: All models exhibit a "performance cliff" when moving from single-hop to multi-hop tasks. For instance, AttH's MRR dropped from 0.778 to 0.218 on the instanceOf task when multi-hop reasoning was required.
  4. Synergy of Joint Modeling: Models generally perform better in the "Joint" setting, indicating that the concept graph provides essential ontological constraints that aid in factual link prediction.

Conclusion

The KACC benchmark reveals that while current models are proficient at simple link prediction, they struggle with the transitive reasoning required for deep hierarchical understanding. Future work should explore contextualized embeddings and the integration of natural language to bridge the gap between symbolic knowledge and commonsense reasoning.

Summary updated

Abstract

A comprehensive knowledge graph (KG) contains an instance-level entity graph and an ontology-level concept graph. The two-view KG provides a testbed for models to "simulate" human's abilities on knowledge abstraction, concretization, and completion (KACC), which are crucial for human to recognize the world and manage learned knowledge. Existing studies mainly focus on partial aspects of KACC. In order to promote thorough analyses for KACC abilities of models, we propose a unified KG benchmark by improving existing benchmarks in terms of dataset scale, task coverage, and difficulty. Specifically, we collect new datasets that contain larger concept graphs, abundant cross-view links as well as dense entity graphs. Based on the datasets, we propose novel tasks such as multi-hop knowledge abstraction (MKA), multi-hop knowledge concretization (MKC) and then design a comprehensive benchmark. For MKA and MKC tasks, we further annotate multi-hop hierarchical triples as harder samples. The experimental results of existing methods demonstrate the challenges of our benchmark. The resource is available at https://github.com/thunlp/KACC.
BibTeX
@inproceedings{zhou2021kacc,
  author = {Jie Zhou and Xin Lv and Cheng Yang and Zhiyuan Liu and Juanzi Li and Maosong Sun},
  title = {{KACC: A Multi-task Benchmark for Knowledge Abstraction, Concretization and Completion}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
  doi = {10.18653/v1/2021.findings-acl.153},
}
Figure