KACC: A Multi-task Benchmark for Knowledge Abstraction, Concretization and Completion

Jie Zhou, Shengding Hu, Xin Lv, Cheng Yang, Zhiyuan Liu, Wei Xu, Jie Jiang, Juanzi Li, Maosong Sun

Abstract

The KACC benchmark introduces a robust framework for evaluating how AI models handle "two-view" Knowledge Graphs—those containing both specific facts (entities) and abstract hierarchies (concepts). By providing large-scale datasets and challenging multi-hop tasks, the researchers reveal that while current models can link entities to concepts, they struggle significantly with deep hierarchical reasoning and "concretization" (finding specific examples for abstract categories).

Introduction: The Abstraction Gap in Knowledge Graphs

Most research in Knowledge Graph (KG) completion focuses on the "entity view"—predicting factual links like (Da Vinci, painted, Mona Lisa). However, human intelligence relies heavily on moving between levels of abstraction. We know that Da Vinci is a painter (abstraction) and that a painter is a type of artist. Conversely, if asked for an example of an artist, we can name Da Vinci (concretization).

Illustration of the two-view Knowledge Graph structure, showing the relationship between the Concept Graph and the Entity Graph.

Existing benchmarks for these "Entity-Concept KGs" (EC-KGs) suffer from three main flaws: scale (concept graphs are often tiny), sparsity (too few links connecting the two views), and simplicity (mostly testing one-hop relationships). The KACC benchmark aims to fill this gap by providing a unified, difficult testbed for these "vertical" reasoning abilities.

Methods and Task Formalization

The authors formalize the EC-KG structure into three primary tasks:

  • Knowledge Abstraction (KA): Predicting the concept an entity belongs to (e.g., Da Vinci $\rightarrow$ Painter).
  • Knowledge Concretization (KC): Predicting an entity that fits a concept (e.g., Painter $\rightarrow$ Da Vinci).
  • Knowledge Completion (KCom): Filling in missing logical links within either the entity or concept graphs.

The most novel contribution is the introduction of Multi-hop versions of these tasks (MKA and MKC). These require models to infer relationships implied by transitivity (e.g., if a Scientist is a Researcher and a Researcher is an Occupation, then a Scientist is an Occupation). Human annotators were employed to filter out "semantic drift" where transitivity fails in real-world logic.

Datasets: Scaling Up Abstraction

The authors released three scales of the benchmark based on Wikidata: KACC-S, KACC-M, and KACC-L.

Statistical comparison of KACC datasets against existing benchmarks YAGO26K-906 and DB111K-174.

As shown in the table above, KACC-M contains roughly 100k entities and 6.6k concepts—38 times more concepts than previous standard datasets like DB111K-174. These datasets feature significantly more cross-view links (averaging >1.09 per entity), ensuring models have enough "bridges" to learn the relationship between facts and abstractions. Furthermore, a topological sort algorithm was used to eliminate cycles in the concept hierarchy, ensuring a clean directed acyclic graph (DAG) structure.

Key Insights and Results

Benchmarking models like TransE, DistMult, and specialized architectures like AttH revealed several critical insights:

  • The "Concretization" Gap: Models are much better at abstracting (Entity $\rightarrow$ Concept) than concretizing (Concept $\rightarrow$ Entity). It is easier for a model to categorize a known object than to retrieve a specific instance from a broad category.
  • Hyperbolic Advantage: Models like AttH, which use hyperbolic geometry, significantly outperform Euclidean models on abstraction tasks. Hyperbolic space is mathematically better suited for representing tree-like, hierarchical structures.
  • The Multi-hop Wall: All existing models suffer a massive performance drop (often over 50% in MRR) when moving from one-hop to multi-hop tasks. This suggests current KG embeddings mostly memorize local neighborhoods rather than understanding transitivity.
  • Joint Modeling: Using both entity and concept graphs simultaneously almost always outperformed single-graph settings, proving that understanding the concept of a person helps a model predict their factual relationships.

Limitations and Future Directions

Currently, no single model excels at both hierarchical reasoning and logical link prediction. Future architectures must balance these distinct types of "thinking." The authors suggest that integrating contextualized embeddings (like BERT) or multimodal data (text/common-sense knowledge) could help bridge the current gap in multi-hop reasoning and address semantic drift.

Summary updated

BibTeX
@inproceedings{zhou2021kacc,
  author = {Jie Zhou and Shengding Hu and Xin Lv and Cheng Yang and Zhiyuan Liu and Wei Xu and Jie Jiang and Juanzi Li and Maosong Sun},
  title = {{KACC: A Multi-task Benchmark for Knowledge Abstraction, Concretization and Completion}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
  doi = {10.18653/v1/2021.findings-acl.153},
}
Figure