Abstract
In this paper, we propose ChronoBias, a novel benchmark for evaluating time-conditional group bias in the time-sensitive knowledge of large language models (LLMs). Our benchmark is constructed via a template-based semi-automated generation method, balancing the quality-quantity trade-off in existing benchmark curation approaches. For knowledge that changes over time, time-conditional group bias exhibits varying patterns across time intervals, evident in both the best- and worst-performing groups and in the bias metric itself. In addition to parametric knowledge bias, we identify time-sensitivity bias as an additional factor after a model's knowledge cutoff, accounting for much of the variation in time-conditional group bias over time. While Retrieval-Augmented Generation (RAG) improves overall performance, the disparate patterns of time-conditional group bias persist, highlighting the need for more accurate and fair RAG-based LLMs.
The Problem: Static Metrics for Dynamic Knowledge
Most fairness benchmarks for LLMs measure "group bias" as a single, static number—for example, checking if a model is better at answering questions about the US than about Algeria. However, knowledge is not static. Existing evaluations are often time-marginalized: they average performance over all time points, ignoring the fact that a model’s accuracy and bias can fluctuate wildly depending on whether the information is historical or occurred after the model's "knowledge cutoff."
The core challenge is that some groups have more "volatile" knowledge than others. In the sports world, a league table in one country might be very predictable (slow-changing), while another is highly chaotic (fast-changing). If an LLM struggles more with fast-changing information, it will naturally appear more biased against the "volatile" group, a phenomenon the authors call time-conditional group bias.
Constructing the ChronoBias Benchmark
To address these issues, the researchers developed a semi-automated pipeline to build the ChronoBias benchmark, focusing on the domain of international football (soccer) league data from 2015 to 2025.
Methodology Highlights:
- Taxonomy of Change: Knowledge is categorized into three levels: Never-changing (e.g., league founding dates), Slow-changing (e.g., season top scorers), and Fast-changing (e.g., current standings).
- Template-based Generation: Structured web-crawled data is converted into QA pairs using templates, ensuring both high quality and large scale (over 13,000 fast-changing questions).
- IoU-based Metrics: Since sports answers often involve lists, the benchmark uses Intersection over Union (IoU) metrics to measure accuracy in both content and ranking.
Key Insights: The Impact of Time-Sensitivity
The evaluation of models like GPT-4o and Llama-3 across different time intervals—Before, Right before, Right after, and After the knowledge cutoff—revealed critical patterns in bias.
The "Brazil Effect" and Parametric Bias
Before the knowledge cutoff, models often perform well on popular regions (like Brazil) due to high data density in pre-training. However, after the cutoff, performance for these volatile regions plummets more drastically than for others. This proves that time-sensitivity bias is a distinct factor from parametric bias (what the model simply didn't see enough of during training).
RAG is Not a Silver Bullet
While Retrieval-Augmented Generation (RAG) improves overall accuracy, it does not fix the underlying pattern of bias. Models still show disparate performance across groups because the quality of retrieved information varies by region. Furthermore, high-capacity models sometimes suffer from "forecasting conflicts," where they prioritize internal hallucinations based on historical trends over current information provided in retrieved documents.
Conclusion and Future Work
ChronoBias demonstrates that bias amplification occurs significantly after the knowledge cutoff. The researchers suggest that future work should focus on "fair-by-design" RAG algorithms. A key takeaway is that retrievers should ideally be inversely biased to the LLM to compensate for the model's internal weaknesses in specific temporal or geographical contexts.
Summary updated