ChronoBias: A Benchmark for Evaluating Temporal Group Bias in the Time-sensitive Knowledge of Large Language Models

Kyungmin Kim, Youngbin Choi, Hyounghun Kim, Dongwoo Kim, Sangdon Park2025 · DOI

Abstract

In this paper, we propose ChronoBias, a novel benchmark for evaluating time-conditional group bias in the time-sensitive knowledge of large language models (LLMs). Our benchmark is constructed via a template-based semi-automated generation method, balancing the quality-quantity trade-off in existing benchmark curation approaches. For knowledge that changes over time, time-conditional group bias exhibits varying patterns across time intervals, evident in both the best- and worst-performing groups and in the bias metric itself. In addition to parametric knowledge bias, we identify time-sensitivity bias as an additional factor after a model's knowledge cutoff, accounting for much of the variation in time-conditional group bias over time. While Retrieval-Augmented Generation (RAG) improves overall performance, the disparate patterns of time-conditional group bias persist, highlighting the need for more accurate and fair RAG-based LLMs.

The Problem: Static Metrics for Dynamic Knowledge

Most fairness benchmarks for LLMs measure "group bias" as a single, static number—for example, checking if a model is better at answering questions about the US than about Algeria. However, knowledge is not static. Existing evaluations are often time-marginalized: they average performance over all time points, ignoring the fact that a model’s accuracy and bias can fluctuate wildly depending on whether the information is historical or occurred after the model's "knowledge cutoff."

The core challenge is that some groups have more "volatile" knowledge than others. In the sports world, a league table in one country might be very predictable (slow-changing), while another is highly chaotic (fast-changing). If an LLM struggles more with fast-changing information, it will naturally appear more biased against the "volatile" group, a phenomenon the authors call time-conditional group bias.

Constructing the ChronoBias Benchmark

To address these issues, the researchers developed a semi-automated pipeline to build the ChronoBias benchmark, focusing on the domain of international football (soccer) league data from 2015 to 2025.

Taxonomy of change and automatic generation pipeline

Methodology Highlights:

  1. Taxonomy of Change: Knowledge is categorized into three levels: Never-changing (e.g., league founding dates), Slow-changing (e.g., season top scorers), and Fast-changing (e.g., current standings).
  2. Template-based Generation: Structured web-crawled data is converted into QA pairs using templates, ensuring both high quality and large scale (over 13,000 fast-changing questions).
  3. IoU-based Metrics: Since sports answers often involve lists, the benchmark uses Intersection over Union (IoU) metrics to measure accuracy in both content and ranking.

Key Insights: The Impact of Time-Sensitivity

The evaluation of models like GPT-4o and Llama-3 across different time intervals—Before, Right before, Right after, and After the knowledge cutoff—revealed critical patterns in bias.

Fluctuation of group bias relative to the model's knowledge cutoff

The "Brazil Effect" and Parametric Bias

Before the knowledge cutoff, models often perform well on popular regions (like Brazil) due to high data density in pre-training. However, after the cutoff, performance for these volatile regions plummets more drastically than for others. This proves that time-sensitivity bias is a distinct factor from parametric bias (what the model simply didn't see enough of during training).

RAG is Not a Silver Bullet

While Retrieval-Augmented Generation (RAG) improves overall accuracy, it does not fix the underlying pattern of bias. Models still show disparate performance across groups because the quality of retrieved information varies by region. Furthermore, high-capacity models sometimes suffer from "forecasting conflicts," where they prioritize internal hallucinations based on historical trends over current information provided in retrieved documents.

Conclusion and Future Work

ChronoBias demonstrates that bias amplification occurs significantly after the knowledge cutoff. The researchers suggest that future work should focus on "fair-by-design" RAG algorithms. A key takeaway is that retrievers should ideally be inversely biased to the LLM to compensate for the model's internal weaknesses in specific temporal or geographical contexts.

Summary updated

Abstract

In this paper, we propose ChronoBias, a novel benchmark for evaluating time-conditional group bias in the time-sensitive knowledge of large language models (LLMs).Our benchmark is constructed via a template-based semiautomated generation method, balancing the quality-quantity trade-off in existing benchmark curation approaches.For knowledge that changes over time, time-conditional group bias exhibits varying patterns across time intervals, evident in both the best-and worst-performing groups and in the bias metric itself.In addition to parametric knowledge bias-which influences group bias across all time intervalswe identify time-sensitivity bias as an additional factor after a model's knowledge cutoff, accounting for much of the variation in timeconditional group bias over time.Since both biases are irreducible, retrieval-augmented generation (RAG) can be a promising approach, as it can address post-cutoff knowledge and better leverage pretraining knowledge that is underrepresented in the model parameters.While RAG improves both overall performance and group bias, we observe that the disparate patterns of time-conditional group bias still persist.Therefore, through extensive experiments with various model configurations, we illustrate how accurate and fair RAG-based LLMs should behave and provide actionable guidelines toward constructing such ideal models. BeforeRight before Right after After 0.0 0.1 0.2 0.3 0.4 0.5 0.
BibTeX
@misc{kim2025chronobias,
  author = {Kyungmin Kim and Youngbin Choi and Hyounghun Kim and Dongwoo Kim and Sangdon Park},
  title = {{ChronoBias: A Benchmark for Evaluating Temporal Group Bias in the Time-sensitive Knowledge of Large Language Models}},
  year = {2025},
  doi = {10.18653/v1/2025.findings-emnlp.405},
}
Figure