Understanding Data Temporality Impact on Large Language Models Pre-training

Romain Fabre, Hippolyte Pilchen, Franck Signe Talla, Patrick Perez, Edouard Grave

While we often discuss a Large Language Model's "knowledge cutoff," the reality of how these models perceive time is much more complex. Even if an LLM is trained on data up to 2024, it frequently performs better on facts from 2018 than from 2023. This phenomenon, known as the knowledge horizon gap, highlights a fundamental flaw in how we pre-train foundational models.

Recent research demonstrates that sequential pre-training—feeding data to the model in strict chronological order—can significantly improve an LLM's "factual freshness" and temporal awareness. By abandoning the standard practice of data shuffling, a 6B-parameter model can actually outperform much larger open-source models on recent events.

The Flaw in Global Shuffling

The primary culprit behind the knowledge horizon gap is global shuffling. By mixing all training data together into a "timeless" soup, researchers implicitly treat the internet as a static pool of information. However, the world is non-stationary; facts change over time (e.g., "Who is the CEO of Twitter?").

Shuffling causes models to prioritize historical information because older facts have been around longer and are repeated across many years of web crawls. This repetition acts as a form of "temporal smoothing" or regression to the mean. The model memorizes older facts more deeply, drowning out newer, more relevant information and creating a temporal delay where the model's effective knowledge lags years behind its actual data cutoff.

Methodology: Chronological Pre-training

To test the impact of data temporality, researchers compared two training regimes for a 6B-parameter Transformer using 2.5 trillion tokens sourced from Common Crawl:

  1. Shuffled Baseline: The standard approach where data from 2020–2024 is randomly mixed.
  2. Sequential Model: The model is trained on web snapshots in strict chronological order from 2018 through 2025.

To ensure a fair comparison, the team employed a branching cooldown strategy. Because models typically perform best right after a learning rate "cooldown" phase, researchers branched off intermediate checkpoints and applied a 30,000-step decay. This allowed them to evaluate each year's performance at its fully converged state.

Datasets and the KairosQA Benchmark

Evaluating temporal knowledge requires specialized tools. To that end, the researchers introduced KairosQA, a new benchmark containing over 7,000 temporally sensitive questions derived from Wikidata. It specifically targets facts that changed at least twice between 2018 and 2025, such as sports team memberships or political office holders.

The models were also evaluated on OLMES (for general language understanding) to ensure sequential training didn't degrade core capabilities, and TAQA, an existing temporal benchmark based on Wikipedia tables.

Key Findings

  • The "Recency Peak": Sequential models consistently achieve their highest accuracy in the year immediately preceding their training endpoint. Shuffled models, conversely, peak on older data.
  • Factual Freshness: The sequential model demonstrated a 16% relative F1 gain on KairosQA for the 2023–2024 period compared to other state-of-the-art open-source models.
  • Scale is Not the Solution: The authors benchmarked heavyweights like Llama 3.1-8B and Qwen3-14B. Despite their massive size, these models showed the same "temporal decay" pattern as the shuffled baseline. A smaller model trained sequentially proved smarter about recent events than models twice its size trained via global shuffling.
  • No Performance Penalty: After processing 2.5T tokens, the sequential model matched or slightly exceeded the shuffled baseline on general language benchmarks. Chronological ordering alters the learning trajectory but does not limit the model's final capacity.

Limitations and the Path Forward

The primary trade-off of sequential training is catastrophic forgetting. As the sequential model learns 2024 facts, it inevitably begins to "forget" the specific details of 2018.

The authors explored two mitigation strategies:

  1. Model Soup: Merging weights from different yearly checkpoints. This resulted in "average" performance rather than a best-of-all-worlds model.
  2. Data Replay: Mixing a small amount of old data (from 2020) into the final training stages. This provided slight improvements but ultimately created a zero-sum trade-off between old and new knowledge.

While sequential ordering is a massive step forward for creating truly up-to-date LLMs, the field still requires new architectures. The ultimate goal is to develop models that can explicitly track the temporal origin of data, updating their understanding of the present without erasing their knowledge of the past.

Summary updated

Abstract

Large language models (LLMs) are typically trained on shuffled corpora, yielding models whose knowledge is frozen at training time and whose temporal grounding remains poorly understood. In this work, we study the impact of pretraining dynamics on the acquisition of time-sensitive factual knowledge, focusing specifically on data ordering. Our main contributions are twofold. First, we introduce a comprehensive benchmark of over 7,000 temporally grounded questions and an evaluation protocol that enables analysis of whether models correctly associate facts with their corresponding time periods. Second, we pretrain 6B-parameter language models on temporally ordered Common Crawl snapshots and compare them against standard shuffled pretraining. Our results show that sequentially trained models match shuffled baselines on general language understanding and common knowledge while consistently exhibiting more up-to-date and temporally precise knowledge. Temporally ordered pretraining yields improved factual freshness, while shuffled pretraining peaks on older data, possibly due to increased factual repetition. These findings, along with the release of our checkpoints and datasets, provide a foundation for future research on continual learning for large language models.
BibTeX
@inproceedings{fabre2026understanding,
  author = {Romain Fabre and Hippolyte Pilchen and Franck SIGNE TALLA and Patrick Perez and Edouard Grave},
  title = {{Understanding Data Temporality Impact on Large Language Models Pre-training}},
  booktitle = {International Conference on Machine Learning},
  year = {2026},
  url = {https://openreview.net/forum?id=P82s05mnUY},
}
Figure