ONSEP: A Novel Online Neural-Symbolic Framework for Event Prediction Based on Large Language Model

Xuanqing Yu, Wangtao Sun, Jingwei Li, Kang Liu, Chengbao Liu, Jie Tan

TL;DR

ONSEP is a new neural-symbolic framework that enhances event prediction in temporal knowledge graphs by dynamically mining causal rules and integrating long-term historical context. By allowing Large Language Models (LLMs) to learn from experience during testing and bridge large temporal gaps, it achieves significant performance gains over standard in-context learning without requiring expensive retraining.

What Problem Does This Paper Address?

Predicting future events—such as political interactions or financial shifts—is often modeled using Temporal Knowledge Graph Forecasting (TKGF). The goal is to predict missing entities in a quadruple like (subject, relation, ?, time).

Current approaches fall into two camps: traditional embedding-based models (GNNs/RNNs) that require resource-intensive training on specific datasets, and newer LLM-based In-Context Learning (ICL) methods. While ICL is adaptable, it suffers from two major flaws:

  1. Short-term Memory: LLMs are limited by their context window, often missing long-term causal trends (e.g., a "phone call" in January causing a "diplomatic visit" in April).
  2. Static Logic: They don't "learn" from their mistakes or successes during the testing phase, failing to adapt to evolving data distributions.

Methods

The authors introduce the Online Neural-Symbolic Event Prediction (ONSEP) framework, which combines the reasoning power of LLMs with a symbolic rule-based system. As shown in Figure 2, the framework consists of two main phases:

Figure 2: Detailed structure of ONSEP framework with two phases: (1) Dynamic causal rule mining(§ 3.1) and (2) dual history augmented generation(§ 3.2). Specially, the DCRM phase employs a semantic-driven algorithm(§ 3.1.1) to identify causal rules and dynamically updates the rule base(§ 3.1.2), incorporating a filtering and sorting mechanism(§ 3.1.3). (2) The DHAG phase utilizes a long short-term bi-branch retriever(§ 3.2.1) alongside a hybrid model inference (§ 3.2.2) strategy to improve prediction accuracy.
  1. Dynamic Causal Rule Mining (DCRM): This module acts as the "symbolic" learner. When the system receives feedback on a prediction, it uses the LLM to identify potential cause-effect links. It then constructs and maintains a Causal Rule Base (CRB). These rules (e.g., Consultation is often preceded by a Phone Call) are updated with confidence scores in real-time, allowing the model to adapt to new patterns without retraining.

  2. Dual History Augmented Generation (DHAG): To overcome context limits, ONSEP uses a bi-branch retriever. One branch pulls the short-term history (the most recent events), while the other pulls a long-term causal chain based on the rules in the CRB. A Hybrid Model Inference (HMI) strategy then merges these two perspectives using a weighted ensemble to make the final prediction.

Datasets

The framework was evaluated on three standard benchmarks from the Integrated Crisis Early Warning System (ICEWS):

  • ICEWS14: Events from 2014.
  • ICEWS05-15: A larger span from 2005 to 2015.
  • ICEWS18: Events from 2018.

These datasets are particularly challenging because they involve real-world political shifts; for instance, ICEWS18 contains entities like Donald Trump that do not appear in the earlier ICEWS14 data, testing the model's ability to handle distribution shifts.

Key Insights and Ideas

  • Self-Improvement via Feedback: Unlike standard LLM inference, ONSEP gets smarter as it processes more test data. By mining rules from its own successes and failures, it builds a domain-specific logic engine on the fly.
  • Causality Bridges Time: The "aha moment" here is that while an LLM can't read 1,000 historical events due to context limits, it can read a few highly relevant "causal" events from months ago if a symbolic rule points it there.
  • Inductive Generalization: Causal rules learned on one dataset (ICEWS14) were shown to improve performance on a completely different one (ICEWS18). This suggests that the underlying "logic" of political events is somewhat universal, even if the specific actors change.
  • Balancing Act: The authors found that a small weight (λ ≈ 0.1) for long-term causal chains is optimal. Too much reliance on long-term rules can introduce noise, but ignoring them entirely (λ = 0) significantly degrades performance, as seen in Figure 3.
Figure 3: Performance of ONSEP in terms of Hit@1 across various DHAG ensemble weights λ of DHAG. The underlying LLM is InternLM2-7B, processing input histories of length 100. λ represents the weight given to long-term causal event chains. This illustrates how varying λ influences the integration of short-term and long-term reasoning contexts within ONSEP.

Key Results

ONSEP consistently outperformed standard ICL across all datasets and metrics (Hit@1, Hit@3, Hit@10).

  • Hit@1 Improvements: Using the InternLM2-7B model, ONSEP improved Hit@1 scores by 9.63% on ICEWS14 and a staggering 16.28% on ICEWS18 compared to standard ICL.
  • Competitive with Supervised Models: Despite being a "training-free" LLM approach, ONSEP achieved results competitive with state-of-the-art embedding-based models like RE-GCN and TITer, which require extensive training on the target data.

Detailed comparisons are provided in the paper's main results table:

Table 2: Performance comparison among LLM-based ICL and traditional embedding-based methods on three datasets with time-aware metrics (Hit@k). The highest performance is highlighted in bold. ∆Improve represents the percentage improvement of ONSEP over ICL. The results of the embedding-based models are excerpted from (Li et al., 2022).

Limitations and Future Work

  • Computational Overhead: Because ONSEP requires multiple LLM calls (one for rule mining/assessment and others for dual-branch inference), it is roughly 2x slower than standard ICL.
  • Interpretability: While the causal rules are symbolic and readable, the final "fusion" step in the LLM remains a black box, making it difficult to trace the exact reasoning path for every prediction.
  • Semantic Dependency: The method relies heavily on the LLM's ability to understand the semantics of relations. It may struggle with datasets that use abstract or non-linguistic identifiers.
  • Future Directions: The authors suggest exploring more efficient indexing for cue words and investigating how this framework might apply to other time-sensitive domains like financial forecasting or recommendation systems.

Summary updated

Abstract

In the realm of event prediction, temporal knowledge graph forecasting (TKGF) stands as a pivotal technique. Previous approaches face the challenges of not utilizing experience during testing and relying on a single short-term history, which limits adaptation to evolving data. In this paper, we introduce the Online Neural-Symbolic Event Prediction (ONSEP) framework, which innovates by integrating dynamic causal rule mining (DCRM) and dual history augmented generation (DHAG). DCRM dynamically constructs causal rules from real-time data, allowing for swift adaptation to new causal relationships. In parallel, DHAG merges short-term and long-term historical contexts, leveraging a bi-branch approach to enrich event prediction. Our framework demonstrates notable performance enhancements across diverse datasets, with significant Hit@k (k=1,3,10) improvements, showcasing its ability to augment large language models (LLMs) for event prediction without necessitating extensive retraining. The ONSEP framework not only advances the field of TKGF but also underscores the potential of neural-symbolic approaches in adapting to dynamic data environments.
BibTeX
@inproceedings{yu2024onsep,
  author = {Xuanqing Yu and Wangtao Sun and Jingwei Li and Kang Liu and Chengbao Liu and Jie Tan},
  title = {{ONSEP: A Novel Online Neural-Symbolic Framework for Event Prediction Based on Large Language Model}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
  doi = {10.18653/v1/2024.findings-acl.378},
}
Figure