DTELS: Towards Dynamic Granularity of Timeline Summarization

Chenlong Zhang, Tong Zhou, Pengfei Cao, Zhuoran Jin, Yubo Chen, Kang Liu, Jun Zhao

Abstract: The rapid proliferation of online news has posed significant challenges in tracking the continuous development of news topics. Traditional timeline summarization constructs a chronological summary of the events but often lacks the flexibility to meet the diverse granularity needs. To overcome this limitation, we introduce a new paradigm, Dynamic-granularity TimELine Summarization, (DTELS), which aims to construct adaptive timelines based on user instructions or requirements. This paper establishes a comprehensive benchmark for DTLES that includes: (1) an evaluation framework grounded in journalistic standards to assess the timeline quality across four dimensions: Informativeness, Granular Consistency, Factuality, and Coherence; (2) a large-scale, multi-source dataset with multiple granularity timeline annotations based on a consensus process to facilitate authority; (3) extensive experiments and analysis with two proposed solutions based on Large Language Models (LLMs) and existing state-of-the-art TLS methods. The experimental results demonstrate the effectiveness of LLM-based solutions. However, even the most advanced LLMs struggle to consistently generate timelines that are both informative and granularly consistent, highlighting the challenges of the DTELS task.

Introduction: Beyond One-Size-Fits-All Timelines

As the volume of online news continues to grow, users increasingly rely on Timeline Summarization (TLS) to track the evolution of complex events. However, traditional TLS systems are typically static, offering a fixed level of detail that may not suit all audiences. For instance, a network engineer investigating a series of data breaches requires a high-resolution, technical timeline, whereas a casual investor may only need a high-level overview of the financial consequences.

To address this, we propose Dynamic-granularity TimELine Summarization (DTELS). Unlike traditional systems, DTELS generates adaptive timelines based on specific user instructions regarding granularity. This shift from static to dynamic summarization ensures that the information provided is relevant to the user's specific context and needs.

Comparison between traditional Timeline Summarization (TLS) and the proposed Dynamic-granularity TimELine Summarization (DTELS).

The "Mount-then-Measure" Evaluation Framework

Evaluating dynamic timelines is inherently difficult because standard metrics like ROUGE—which rely on word overlap—fail to capture structural nuances like event gaps or factual consistency. We introduce a novel "Mount-then-Measure" framework that decomposes timelines into "event atoms" (the smallest units of distinguishable information).

Using a Hungarian algorithm to align predicted nodes with reference nodes, we evaluate timelines across four critical dimensions:

  1. Informativeness: Measures the coverage of key event atoms.
  2. Granular Consistency: Assesses whether the temporal "jumps" and level of detail match the user's requested granularity.
  3. Factuality: Ensures every atom can be verified against the source articles, penalizing hallucinations.
  4. Coherence: Evaluates the logical flow, linguistic clarity, and lack of redundancy using a specialized rubric.
The four-dimensional evaluation framework for DTELS: Informativeness, Granular Consistency, Factuality, and Coherence.

DTELS-Bench and LLM Strategies

To support this new paradigm, we developed DTELS-Bench, a large-scale Chinese dataset comprising 543 news topics and over 55,000 articles. Unlike previous benchmarks, DTELS-Bench provides human-refined reference timelines at three distinct levels: Fine, Medium, and Coarse.

We benchmarked several Large Language Model (LLM) strategies against traditional extractive methods:

  • Long-context Prompting (LP): Utilizing models with massive context windows (up to 200k tokens) to process entire news corpora at once.
  • Hierarchical Merging (HM): A recursive approach that summarizes daily news and then merges those summaries into a final timeline, suitable for models with smaller context windows.

Key Insights and Results

Our experiments reveal that while LLMs significantly outperform traditional baselines, they face a persistent consistency-detail trade-off. As models attempt to increase informativeness by adding more details, they often struggle to maintain the requested granularity, sometimes slipping into overly fine-grained reporting when a coarse overview was requested.

Other key findings include:

  • Human Alignment: Our proposed metrics show a high correlation with human judgment, particularly for coherence (99.14% Pearson correlation).
  • Scale Matters: Larger models, such as Qwen-110B, demonstrate a superior ability to distinguish between salient and minor events, suggesting that understanding event importance is an emergent property of model scale.
  • Topic Sensitivity: Performance varies by topic; for example, military news tends to be more structured and easier for models to summarize than the nuanced shifts in political or technological landscapes.

Conclusion

DTELS represents a significant step toward personalized information retrieval. While LLMs show great promise in generating adaptive timelines, the challenge of balancing informativeness with granular consistency remains. Future work will focus on expanding DTELS-Bench to more languages and developing more efficient evaluation methods that reduce reliance on expensive API calls while maintaining high accuracy.

Summary updated

Abstract

The rapid proliferation of online news has posed significant challenges in tracking the continuous development of news topics. Traditional timeline summarization constructs a chronological summary of the events but often lacks the flexibility to meet the diverse granularity needs. To overcome this limitation, we introduce a new paradigm, Dynamic-granularity TimELine Summarization, (DTELS), which aims to construct adaptive timelines based on user instructions or requirements. This paper establishes a comprehensive benchmark for DTLES that includes: (1) an evaluation framework grounded in journalistic standards to assess the timeline quality across four dimensions: Informativeness, Granular Consistency, Factuality, and Coherence; (2) a large-scale, multi-source dataset with multiple granularity timeline annotations based on a consensus process to facilitate authority; (3) extensive experiments and analysis with two proposed solutions based on Large Language Models (LLMs) and existing state-of-the-art TLS methods. The experimental results demonstrate the effectiveness of LLM-based solutions. However, even the most advanced LLMs struggle to consistently generate timelines that are both informative and granularly consistent, highlighting the challenges of the DTELS task.
BibTeX
@inproceedings{zhang2025dtels,
  author = {Chenlong Zhang and Tong Zhou and Pengfei Cao and Zhuoran Jin and Yubo Chen and Kang Liu and Jun Zhao},
  title = {{DTELS: Towards Dynamic Granularity of Timeline Summarization}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2025},
  doi = {10.18653/v1/2025.naacl-long.136},
}
Figure