Timeline-based Sentence Decomposition with In Context Learning for Temporal Fact Extraction

Jianhao Chen, Haoyuan Ouyang, Junyang Ren, Wentao Ding, Wei Hu, Yuzhong Qu

Introduction: The Challenge of Temporal Fact Extraction

Knowledge Graphs (KGs) serve as the backbone for many modern AI applications. However, traditional KGs often lack a critical temporal dimension. While knowing "Michael Jordan played for the Chicago Bulls" is valuable, knowing when he played for them is essential for complex reasoning and historical accuracy.

The task of Temporal Fact Extraction aims to extract quintuples—consisting of a head entity, relation, tail entity, time qualifier, and time value—directly from unstructured text. Natural language, however, is often complex and "succinct." A single sentence may interweave multiple entities, dates, and events. Existing models frequently struggle with the correspondence problem: while they can identify entities and dates, they often fail to correctly link specific events to their corresponding times when multiple facts are present in one sentence.

Example of complex temporal fact extraction where multiple players and years are interwoven.

Methodology: Timeline-based Sentence Decomposition (TSD)

To address these challenges, researchers developed TSDRE, a framework that leverages the reasoning capabilities of Large Language Models (LLMs) to simplify complex inputs. The core of this approach is Timeline-based Sentence Decomposition (TSD), a two-stage process:

  1. LLM-Driven Decomposition: Using ChatGPT-3.5 with In-Context Learning (ICL), the system acts as a reasoning engine. Instead of performing direct extraction, the LLM decomposes complex sentences into a chronological list of simple, explicit events. To enhance accuracy, the authors utilized a human feedback loop, incorporating "negative examples" into the prompts to steer the model away from common logical errors.
  2. TSDRE Framework: The original text, augmented by the LLM-generated timeline, is used to fine-tune smaller, more efficient Pre-trained Language Models (PLMs) such as Flan-T5 or Llama2 (via LoRA). This division of labor allows the LLM to handle high-level logical "unfolding" while the smaller model focuses on precise extraction according to specific relational schemas.
The TSDRE framework comparing standard extraction (Low Recall) with timeline-augmented extraction (High Recall).

Datasets and Evaluation

The framework was evaluated on two primary datasets:

  • HyperRED-Temporal: A filtered version of a standard hyper-relational dataset.
  • ComplexTRED: A new, challenging benchmark introduced by the authors. It contains 19,148 sentences characterized by high density—averaging 3.46 time expressions and 2.10 facts per sentence. This dataset was constructed using distant supervision from Wikipedia and DBpedia, followed by rigorous manual refinement.

Key Results and Insights

The study yielded several critical insights into the nature of temporal extraction:

  • State-of-the-Art Performance: TSDRE (using Flan-T5-Large) achieved an F1 score of 66.71 on HyperRED-Temporal and 42.55 on the complex ComplexTRED, significantly outperforming previous state-of-the-art models like CubeRE.
  • LLMs as Reasoners, Not Extractors: Interestingly, while ChatGPT-3.5 excelled at logical decomposition, its raw extraction performance was relatively low. This suggests that LLMs are better suited for structural reasoning than for adhering to the rigid, high-cardinality label schemas required for KG extraction.
  • Synergy of Scale: The best results came from combining the reasoning power of large models with the fine-tuning precision of smaller models. Adding timeline decomposition to a Llama2-7B model, for instance, resulted in an 11-point F1 score improvement.

Conclusion and Future Directions

TSDRE demonstrates that teaching models "when" an event occurred requires more than just pattern matching; it requires structural reasoning. By decomposing sentences into timelines, the framework effectively "de-noises" complex syntax. Future work will look toward extending this decomposition to the document level and addressing the nuances of relative time (e.g., "two weeks later"), which remains a significant hurdle in the field.

Summary updated

Abstract

Facts extraction is pivotal for constructing knowledge graphs. Recently, the increasing demand for temporal facts in downstream tasks has led to the emergence of the task of temporal fact extraction. In this paper, we specifically address the extraction of temporal facts from natural language text. Previous studies fail to handle the challenge of establishing time-to-fact correspondences in complex sentences. To overcome this hurdle, we propose a timeline-based sentence decomposition strategy using large language models (LLMs) with in-context learning, ensuring a fine-grained understanding of the timeline associated with various facts. In addition, we evaluate the performance of LLMs for direct temporal fact extraction and get unsatisfactory results. To this end, we introduce TSDRE, a method that incorporates the decomposition capabilities of LLMs into the traditional fine-tuning of smaller pre-trained language models (PLMs). To support the evaluation, we construct ComplexTRED, a complex temporal fact extraction dataset. Our experiments show that TSDRE achieves state-of-the-art results on both HyperRED-Temporal and ComplexTRED datasets.
BibTeX
@inproceedings{chen2024timeline,
  author = {Jianhao Chen and Haoyuan Ouyang and Junyang Ren and Wentao Ding and Wei Hu and Yuzhong Qu},
  title = {{Timeline-based Sentence Decomposition with In Context Learning for Temporal Fact Extraction}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
  doi = {10.18653/v1/2024.acl-long.187},
}
Figure