HTCCN: Temporal Causal Convolutional Networks with Hawkes Process for Extrapolation Reasoning in Temporal Knowledge Graphs

Tingxuan Chen, Jun Long, Liu Yang, Zidong Wang, Yongheng Wang, Xiongnan JinNAACL 2024 · DOI

TL;DR

This paper introduces HTCCN, a novel model for predicting future facts in Temporal Knowledge Graphs (TKGs) by effectively modeling the complex temporal dynamics and interdependencies of historical facts. HTCCN uniquely combines Temporal Causal Convolutional Networks (TCCN) with the Hawkes process to capture both individual fact evolution and collective node behaviors, leading to state-of-the-art performance in extrapolation reasoning.

What Problem Does This Paper Address?

Temporal Knowledge Graphs (TKGs) are powerful tools for representing dynamic information, where facts evolve over time and influence each other. A key challenge in TKG research is extrapolation reasoning: predicting future facts based on historical data. Existing methods often struggle with this because they either oversimplify temporal dynamics (e.g., assuming equidistant time intervals) or fail to adequately capture the intricate causal relationships between facts. This can lead to information loss and inaccurate predictions, especially when dealing with complex, real-world temporal patterns. The paper aims to bridge this gap by developing a model that can accurately reason about future events by deeply understanding the historical evolution of facts and their interconnections.

Methods

HTCCN is built upon three core components:

  • Temporal Causal Convolutional Network (TCCN): This module is designed to capture historical fact dependencies. It uses dilated causal convolutions, which allow the network to look back at a wider range of historical information without increasing the number of layers. This helps in modeling complex temporal dynamics over diverse time intervals.
  • Hawkes Process Integration: The Hawkes process is a point process model that excels at capturing self-exciting phenomena, where the occurrence of an event increases the probability of future events. In HTCCN, it's used to model the "link formation processes" within TKGs, essentially quantifying how historical facts influence the likelihood of new facts forming. This is achieved by defining a conditional intensity function that captures the excitement induced by past events.
  • Dual-Level Dynamics: HTCCN incorporates two levels of dynamics for a comprehensive understanding of TKG evolution:
  • Fact-level dynamics: Focuses on individual fact characteristics, including conditional intensities, and the prediction of entities and relations.
  • Node-level dynamics: Captures the collective behavior of nodes, considering their "fact tendencies" and how they tend to form new links over time. This is modeled using a node dynamics estimator.

These components are integrated into a unified framework with a joint training objective that balances fact-level and node-level losses.

Datasets

The paper evaluates HTCCN on four widely recognized real-world TKG datasets:

  • ICEWS14
  • ICEWS05-15
  • ICEWS18
  • GDELT

These datasets represent historical event data and are commonly used benchmarks in TKG research. No new datasets were introduced in this paper.

Key Insights and Ideas

  • Hawkes Process for Temporal Causality: The paper highlights the effectiveness of the Hawkes process in modeling the "excitement" or influence of past events on future ones in TKGs. This is a crucial insight for understanding the causal nature of temporal knowledge.
  • Dilated Causal Convolutions for Temporal Memory: The use of dilated causal convolutions in TCCN is key to expanding the model's receptive field and capturing long-term temporal dependencies efficiently, overcoming limitations of standard RNNs that might struggle with long sequences.
  • Dual-Level Dynamics for Holistic Understanding: The separation and joint optimization of fact-level and node-level dynamics provide a more nuanced understanding of TKG evolution. Fact-level dynamics capture individual fact importance, while node-level dynamics account for emergent collective behaviors, leading to more robust predictions.
  • Intensity-Guided Decoder for Extrapolation: The proposed Intensity-ConvTransE decoder leverages the learned conditional intensities to guide the prediction process, making it more suitable for extrapolation tasks compared to standard decoders.

Key Results

HTCCN consistently outperforms 15 state-of-the-art baselines across all four benchmark datasets on entity prediction tasks.

  • Average Improvement: HTCCN achieved an average improvement of 2.45% in MRR, 2.39% in Hits@1, and 2.23% in Hits@3 compared to the second-best performing models.
  • Significant Gains: Notably, HTCCN showed substantial improvements in Hits@1, with an increase of approximately 5.27% on GDELT and 2.64% on ICEWS14.
  • Ablation Studies: Ablation studies confirmed the efficacy of each core component:
  • Removing TCCN led to a significant drop in performance, underscoring its role in capturing fact evolution.
  • Removing the Hawkes process also degraded performance, highlighting its importance in modeling fact influence.
  • The intensity-guided decoder showed a notable enhancement over standard decoders for extrapolation.
  • Both fact-level and node-level dynamics were found to be crucial, with fact-level dynamics having a more substantial impact.

Limitations and Future Work

  • Computational Cost: While HTCCN achieves superior performance, it has a relatively higher number of learnable parameters and a larger overall parameter size compared to some baselines. However, the authors note that this is within acceptable limits for modern storage capabilities.
  • Dataset Size and Complexity: The paper focuses on extrapolation reasoning. Future work could explore how HTCCN performs on datasets with even more complex temporal structures or a larger number of entities and relations.
  • Hyperparameter Sensitivity: The paper mentions hyperparameters $\eta_1$ and $\eta_2$ that control the balance between fact-level and node-level dynamics. Further investigation into their optimal tuning across different datasets could be beneficial.
  • Generalizability: While tested on four datasets, exploring HTCCN's performance on a wider variety of TKG domains (e.g., social networks, biological pathways) could demonstrate its broader applicability.

Summary updated

BibTeX
@inproceedings{chen2024htccn,
  author = {Tingxuan Chen and Jun Long and Liu Yang and Zidong Wang and Yongheng Wang and Xiongnan Jin},
  title = {{HTCCN: Temporal Causal Convolutional Networks with Hawkes Process for Extrapolation Reasoning in Temporal Knowledge Graphs}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2024},
  doi = {10.18653/v1/2024.naacl-long.225},
}
Figure