Towards Better Evolution Modeling for Temporal Knowledge Graphs

Zhang Jiasheng, Li Zhangpin, Wang Mingzhe, Shao Jie, Jiangtao Cui, Li Hui

TL;DR

Current models for Temporal Knowledge Graphs (TKGs) are "cheating" by exploiting co-occurrence shortcuts rather than actually learning how facts evolve over time. This paper introduces the TKG Evolution Benchmark, featuring four bias-corrected datasets and two novel tasks—Generative Forecasting and Obsolescence Prediction—to force models to move beyond simple memorization and truly model the dynamics of human knowledge.

What Problem Does This Paper Address?

Temporal Knowledge Graphs (TKGs) store facts that change over time, such as (Lionel Messi, plays for, PSG, 2021-2023). While recent models boast impressive accuracy on standard benchmarks, the authors discovered a startling reality: a "dumb" heuristic that simply counts how often entities appear together (ignoring time entirely) performs nearly as well as state-of-the-art (SOTA) supervised models.

The authors identify three core issues in existing TKG research:

  1. Dataset Bias: High repetition between training and test sets allows models to succeed by memorizing frequent entity pairs.
  2. Oversimplified Tasks: The standard "link prediction" task (predicting a missing object) provides too much "hint" information, making the shortcut easy to exploit.
  3. Neglect of Obsolescence: Most benchmarks focus on when a fact starts but ignore when it ends (obsolescence), which is a critical part of knowledge evolution.

Methods

To break the shortcut, the authors propose a more rigorous approach to TKG modeling:

  • Bias-Corrected Construction: They used k-core filtering to ensure entities have diverse interactions rather than just frequent ones. They also used popularity-based sampling over long time spans to reduce the overlap between training and testing data.
  • Generative Knowledge Forecasting: Instead of "filling in the blank" for a single fact, models must now generate all future facts for a given subject entity. This mimics real-world scenarios like predicting all future actions of a political leader based on history.
  • Knowledge Obsolescence Prediction: A new task where the model must predict when a currently valid fact will become invalid, triggered by the emergence of new, conflicting information.
  • Semantic Enrichment: They replaced cryptic numerical IDs (like Q1070888) with rich, disambiguated textual descriptions (e.g., "Argentine association football player") to allow Large Language Models (LLMs) to leverage their internal world knowledge.

Datasets

The authors introduced the TKG Evolution Benchmark, which includes four new datasets that align timestamp facts with time-interval facts:

  • FinWiki: Financial market data (companies, indices, and market effects).
  • ICEWSWiki: Socio-political events enriched with Wikidata descriptions.
  • YAGO130K: A large-scale general knowledge dataset derived from YAGO.
  • WIKI500K: A massive dataset built from the most popular entities on Wikidata.

These differ from existing benchmarks (like ICEWS14 or GDELT) by offering significantly lower train-test overlap and much richer textual annotations.

Key Insights and Ideas

  • The "Aha" Moment on Shortcuts: The realization that SOTA performance was highly correlated with a non-learnable frequency-counting heuristic suggests that we haven't been measuring "intelligence" or "evolution modeling," but rather "statistical memorization."
  • Evolution is Semantic, Not Just Structural: Traditional embedding-based models struggle when the "shortcut" is removed. However, LLMs (like GPT-4o) show significant gains when provided with textual descriptions, suggesting that understanding the meaning of an entity is crucial for predicting its future.
  • Obsolescence is Triggered: The paper posits that facts don't just expire; they are often "pushed out" by new events. By unifying timestamp and interval data, the benchmark allows researchers to study this causal relationship for the first time.

Key Results

  • Shortcut Eliminated: On the new benchmark, the co-occurrence heuristic performed 10x worse than on old datasets, proving the new datasets successfully closed the "cheating" loophole.
  • SOTA Models Struggle: Even the best traditional TKG models (like HGE and TNT) performed poorly on the Generative task, often achieving Recall@50 scores below 15%. This highlights a massive gap in current modeling capabilities.
  • LLMs Lead in Obsolescence: LLM-based models (TKG-ICL) outperformed traditional methods in predicting when facts end, provided they had access to textual descriptions.
  • Inductive Strength: LLMs handled "unseen" entities (entities not present in training) much better than embedding-based models, which typically fail completely in such scenarios.

Limitations and Future Work

  • LLM Efficiency: While LLMs are better at semantic reasoning, they are computationally expensive and slow compared to lightweight embedding models, making them difficult to deploy for massive, real-time TKGs.
  • Granularity Issues: Some datasets (like Wikidata) use year-level granularity, which can obscure fast-moving causal relationships.
  • Future Directions: The authors suggest developing hybrid models that combine the structural efficiency of graph embeddings with the semantic reasoning power of LLMs to create more robust and scalable evolution models.

Summary updated

Abstract

Temporal knowledge graphs (TKGs) structurally preserve evolving human knowledge. Recent research has focused on designing models to learn the evolutionary nature of TKGs to predict future facts, achieving impressive results. For instance, Hits@10 scores over 0.9 on YAGO dataset. However, we find that existing benchmarks inadvertently introduce a shortcut. Near state-of-the-art performance can be simply achieved by counting co-occurrences, without using any temporal information. In this work, we examine the root cause of this issue, identifying inherent biases in current datasets and over simplified form of evaluation task that can be exploited by these biases. Through this analysis, we further uncover additional limitations of existing benchmarks, including unreasonable formatting of time-interval knowledge, ignorance of learning knowledge obsolescence, and insufficient information for precise evolution understanding, all of which can amplify the shortcut and hinder a fair assessment. Therefore, we introduce the TKG evolution benchmark. It includes four bias-corrected datasets and two novel tasks closely aligned with the evolution process, promoting a more accurate understanding of the challenges in TKG evolution modeling. Benchmark is available at: https://github.com/zjs123/TKG-Benchmark.
BibTeX
@misc{jiasheng2026towards,
  author = {Zhang Jiasheng and Li Zhangpin and Wang Mingzhe and Shao Jie and Jiangtao Cui and Li Hui},
  title = {{Towards Better Evolution Modeling for Temporal Knowledge Graphs}},
  year = {2026},
  eprint = {2602.08353},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2602.08353},
}
Figure