TL;DR
ONSEP is a new neural-symbolic framework that enhances event prediction in temporal knowledge graphs by dynamically mining causal rules and integrating long-term historical context. By allowing Large Language Models (LLMs) to learn from experience during testing and bridge large temporal gaps, it achieves significant performance gains over standard in-context learning without requiring expensive retraining.
What Problem Does This Paper Address?
Predicting future events—such as political interactions or financial shifts—is often modeled using Temporal Knowledge Graph Forecasting (TKGF). The goal is to predict missing entities in a quadruple like (subject, relation, ?, time).
Current approaches fall into two camps: traditional embedding-based models (GNNs/RNNs) that require resource-intensive training on specific datasets, and newer LLM-based In-Context Learning (ICL) methods. While ICL is adaptable, it suffers from two major flaws:
- Short-term Memory: LLMs are limited by their context window, often missing long-term causal trends (e.g., a "phone call" in January causing a "diplomatic visit" in April).
- Static Logic: They don't "learn" from their mistakes or successes during the testing phase, failing to adapt to evolving data distributions.
Methods
The authors introduce the Online Neural-Symbolic Event Prediction (ONSEP) framework, which combines the reasoning power of LLMs with a symbolic rule-based system. As shown in Figure 2, the framework consists of two main phases:
-
Dynamic Causal Rule Mining (DCRM): This module acts as the "symbolic" learner. When the system receives feedback on a prediction, it uses the LLM to identify potential cause-effect links. It then constructs and maintains a Causal Rule Base (CRB). These rules (e.g., Consultation is often preceded by a Phone Call) are updated with confidence scores in real-time, allowing the model to adapt to new patterns without retraining.
-
Dual History Augmented Generation (DHAG): To overcome context limits, ONSEP uses a bi-branch retriever. One branch pulls the short-term history (the most recent events), while the other pulls a long-term causal chain based on the rules in the CRB. A Hybrid Model Inference (HMI) strategy then merges these two perspectives using a weighted ensemble to make the final prediction.
Datasets
The framework was evaluated on three standard benchmarks from the Integrated Crisis Early Warning System (ICEWS):
- ICEWS14: Events from 2014.
- ICEWS05-15: A larger span from 2005 to 2015.
- ICEWS18: Events from 2018.
These datasets are particularly challenging because they involve real-world political shifts; for instance, ICEWS18 contains entities like Donald Trump that do not appear in the earlier ICEWS14 data, testing the model's ability to handle distribution shifts.
Key Insights and Ideas
- Self-Improvement via Feedback: Unlike standard LLM inference, ONSEP gets smarter as it processes more test data. By mining rules from its own successes and failures, it builds a domain-specific logic engine on the fly.
- Causality Bridges Time: The "aha moment" here is that while an LLM can't read 1,000 historical events due to context limits, it can read a few highly relevant "causal" events from months ago if a symbolic rule points it there.
- Inductive Generalization: Causal rules learned on one dataset (ICEWS14) were shown to improve performance on a completely different one (ICEWS18). This suggests that the underlying "logic" of political events is somewhat universal, even if the specific actors change.
- Balancing Act: The authors found that a small weight (λ ≈ 0.1) for long-term causal chains is optimal. Too much reliance on long-term rules can introduce noise, but ignoring them entirely (λ = 0) significantly degrades performance, as seen in Figure 3.
Key Results
ONSEP consistently outperformed standard ICL across all datasets and metrics (Hit@1, Hit@3, Hit@10).
- Hit@1 Improvements: Using the InternLM2-7B model, ONSEP improved Hit@1 scores by 9.63% on ICEWS14 and a staggering 16.28% on ICEWS18 compared to standard ICL.
- Competitive with Supervised Models: Despite being a "training-free" LLM approach, ONSEP achieved results competitive with state-of-the-art embedding-based models like RE-GCN and TITer, which require extensive training on the target data.
Detailed comparisons are provided in the paper's main results table:
Limitations and Future Work
- Computational Overhead: Because ONSEP requires multiple LLM calls (one for rule mining/assessment and others for dual-branch inference), it is roughly 2x slower than standard ICL.
- Interpretability: While the causal rules are symbolic and readable, the final "fusion" step in the LLM remains a black box, making it difficult to trace the exact reasoning path for every prediction.
- Semantic Dependency: The method relies heavily on the LLM's ability to understand the semantics of relations. It may struggle with datasets that use abstract or non-linguistic identifiers.
- Future Directions: The authors suggest exploring more efficient indexing for cue words and investigating how this framework might apply to other time-sensitive domains like financial forecasting or recommendation systems.
Summary updated