While we often discuss a Large Language Model's "knowledge cutoff," the reality of how these models perceive time is much more complex. Even if an LLM is trained on data up to 2024, it frequently performs better on facts from 2018 than from 2023. This phenomenon, known as the knowledge horizon gap, highlights a fundamental flaw in how we pre-train foundational models.
Recent research demonstrates that sequential pre-training—feeding data to the model in strict chronological order—can significantly improve an LLM's "factual freshness" and temporal awareness. By abandoning the standard practice of data shuffling, a 6B-parameter model can actually outperform much larger open-source models on recent events.
The Flaw in Global Shuffling
The primary culprit behind the knowledge horizon gap is global shuffling. By mixing all training data together into a "timeless" soup, researchers implicitly treat the internet as a static pool of information. However, the world is non-stationary; facts change over time (e.g., "Who is the CEO of Twitter?").
Shuffling causes models to prioritize historical information because older facts have been around longer and are repeated across many years of web crawls. This repetition acts as a form of "temporal smoothing" or regression to the mean. The model memorizes older facts more deeply, drowning out newer, more relevant information and creating a temporal delay where the model's effective knowledge lags years behind its actual data cutoff.
Methodology: Chronological Pre-training
To test the impact of data temporality, researchers compared two training regimes for a 6B-parameter Transformer using 2.5 trillion tokens sourced from Common Crawl:
- Shuffled Baseline: The standard approach where data from 2020–2024 is randomly mixed.
- Sequential Model: The model is trained on web snapshots in strict chronological order from 2018 through 2025.
To ensure a fair comparison, the team employed a branching cooldown strategy. Because models typically perform best right after a learning rate "cooldown" phase, researchers branched off intermediate checkpoints and applied a 30,000-step decay. This allowed them to evaluate each year's performance at its fully converged state.
Datasets and the KairosQA Benchmark
Evaluating temporal knowledge requires specialized tools. To that end, the researchers introduced KairosQA, a new benchmark containing over 7,000 temporally sensitive questions derived from Wikidata. It specifically targets facts that changed at least twice between 2018 and 2025, such as sports team memberships or political office holders.
The models were also evaluated on OLMES (for general language understanding) to ensure sequential training didn't degrade core capabilities, and TAQA, an existing temporal benchmark based on Wikipedia tables.
Key Findings
- The "Recency Peak": Sequential models consistently achieve their highest accuracy in the year immediately preceding their training endpoint. Shuffled models, conversely, peak on older data.
- Factual Freshness: The sequential model demonstrated a 16% relative F1 gain on KairosQA for the 2023–2024 period compared to other state-of-the-art open-source models.
- Scale is Not the Solution: The authors benchmarked heavyweights like Llama 3.1-8B and Qwen3-14B. Despite their massive size, these models showed the same "temporal decay" pattern as the shuffled baseline. A smaller model trained sequentially proved smarter about recent events than models twice its size trained via global shuffling.
- No Performance Penalty: After processing 2.5T tokens, the sequential model matched or slightly exceeded the shuffled baseline on general language benchmarks. Chronological ordering alters the learning trajectory but does not limit the model's final capacity.
Limitations and the Path Forward
The primary trade-off of sequential training is catastrophic forgetting. As the sequential model learns 2024 facts, it inevitably begins to "forget" the specific details of 2018.
The authors explored two mitigation strategies:
- Model Soup: Merging weights from different yearly checkpoints. This resulted in "average" performance rather than a best-of-all-worlds model.
- Data Replay: Mixing a small amount of old data (from 2020) into the final training stages. This provided slight improvements but ultimately created a zero-sum trade-off between old and new knowledge.
While sequential ordering is a massive step forward for creating truly up-to-date LLMs, the field still requires new architectures. The ultimate goal is to develop models that can explicitly track the temporal origin of data, updating their understanding of the present without erasing their knowledge of the past.
Summary updated