Most safety leaderboards for language models no longer rely on people to decide whether a model's answer was harmful. An automated judge does that job: a classifier such as Llama Guard, or a strong LLM prompted to grade. That judge is itself a language model with a training cutoff. Harm keeps moving after that cutoff. New coded slurs appear, new events become pretexts for harassment, and new scams attach themselves to new news. We hypothesize that automated safety judges go stale in a measurable way on harms that postdate their training. If so, models tuned to please a fixed judge on a fixed prompt set can look safe while failing on exactly the harms that matter now. The obvious fix is to send those cases back to human annotators. But the literature is clear that reading raw toxic material all day harms people and also degrades their labels.
This note proposes to measure how quickly safety judges go stale on post-cutoff harms. It then tests a specific way to bring humans back only where they are needed, without exposing them to the full toxic item. The judge flags items it cannot ground in its own knowledge. Those items are split into small, separately answerable sub-questions, such as "does this term refer to a group?", "does the response give actionable steps?", and "is this event real and recent?". Each sub-question is shown to a reviewer with slurs masked and retrieved context attached. The core question is empirical: does decomposed, distanced review recover the accuracy of full-exposure human review on stale-judge cases, at a fraction of the exposure? We also map where it breaks, namely on harms that exist only in the combination of parts.
Core Claim
Two claims are tested, and either can fail independently. The first is that automated safety judges show a temporal drop on harms tied to post-cutoff language and events. That drop is large enough to reorder model rankings compared with human-audited labels. Static safety leaderboards would then be partly measuring agreement with a frozen judge, which is a safety analogue of the contamination gap A Careful Examination of Large Language Model Performance on Grade School Arithmetic found by rebuilding GSM8k as GSM1k. The second claim is that a decomposition in the spirit of divide-and-conquer prompting can serve as a human-exposure-reduction tool rather than only a reasoning aid. Divide-and-conquer prompting means solving independent sub-problems in isolated contexts, then merging the answers. The prediction is that it preserves judgment accuracy on items whose harm factors into separable cues, and that it fails predictably on items whose harm is compositional.
Where This Sits
The motivation comes from a tension the file documents directly. On one side, GPT is Not an Annotator shows that GPT-3.5 used as a fairness annotator hallucinates stereotypes and swaps victims with perpetrators. Its scores correlate with human-built ones at only $R^2 = 0.14$ on WinoQueer. Responsible Crowdsourcing for Responsible Generative AI argues that non-expert and minority judgments catch failures that consensus and automation miss. On the other side, Handling and Presenting Harmful Text in NLP Research and Q16 document that exposure causes harm and that tired, desensitized annotators produce inconsistent labels. They also supply partial remedies. HARMCHECK recommends placeholders, grayed text and exposure throttling. Q16 lets curators review machine-flagged images through captions instead of raw pixels. Those remedies operate on whole items. We reuse their distancing techniques but change the unit that a human sees.
The closest systems to this proposal automate red-teaming or keep safety benchmarks fresh. ALERT scores 45,000 static prompts with Llama Guard. MART runs a closed adversary-versus-target loop scored by fixed reward models. Automatic LLM Red Teaming trains a hierarchical reinforcement-learning attacker against LlamaGuard. LiveSecBench comes closest. It renews Chinese-language safety items by expanding human-verified seed intents and grades them with DeepSeek-V3.2, reporting 98.5% agreement with humans. All four treat the judge as fixed ground truth, and none measures what human verification costs the verifiers. We treat the judge's staleness as the object of study. We also treat reviewer exposure as a measured quantity with a budget.
From the reasoning literature we borrow method and a constraint. When Splitting Makes Stronger shows that divide-and-conquer helps when sub-tasks are independent and error-prone. The same paper states that it does not suit tightly serial dependencies. Harm judgment is often not independent: a benign term becomes a slur only next to a particular target. We therefore expect a failure class, and we plan to measure it rather than assume it away. ASTRA shows that a model's intermediate reasoning is where brittleness hides. We use a related idea on the judge side, asking the judge to state which sub-cue it could not verify. The staleness mechanism itself is borrowed from RealTime QA and Mitigating Temporal Misalignment by Discarding Outdated Facts. Both find that models facing post-cutoff content fall back on outdated beliefs instead of abstaining.
Research Program
The first component is a temporally stratified safety test set. We take ALERT's 32 fine-grained risk categories as a skeleton. For each category we write items in three strata. The first is timeless, meaning the same harm expressed in long-standing language. The second is pre-cutoff-event, meaning harm framed around events before the judges' training cutoffs. The third is post-cutoff-event and post-cutoff-lexicon, meaning harm framed around events and coded terms that emerged after the cutoffs. Following RealTime QA's weekly news-sourcing procedure, we collect these from dated news and public moderation reports. Following LiveSecBench's seed-expansion procedure, we expand each seed by varying scenario, target group and execution. Target models then respond to each item, both single-turn and in multi-turn attacks generated with a MART-style adversary. Matching strata on category and surface form, much as GSM1k matched GSM8k on difficulty statistics, lets us attribute judge error to recency rather than to harder content.
The second component is gold labeling. Gold labels come from a small panel of trained moderators who see full items. They work under an IRB protocol with capped session lengths, paid breaks and counseling access. This panel is the unavoidable cost of the study. The design shrinks it to a calibration role instead of making it the pipeline.
The third component is the review pipeline. A judge labels every item and also reports which parts it could not verify, such as an unknown term, an unrecognized event, or unclear intent. Items with high judge uncertainty or unverified recent references are routed to people. Before any person sees an item, an LLM decomposes it into three to six sub-questions, each answerable from a fragment. Slurs and identity terms are masked with HARMCHECK-style placeholders. Retrieved, dated context from a news index is attached to sub-questions about terms or events. Different reviewers answer different sub-questions, so no single reviewer reconstructs the whole item. A fixed merge rule, written before data collection, combines the sub-answers into a harm label.
Exposure is measured in two ways. The first is objective: raw toxic tokens displayed, and unmasked target-group mentions displayed. The second is a brief self-reported distress check at the start and end of each session. Fatigue is measured the way the file's causal chain implies. We track each reviewer's agreement with gold as a function of position within the session. Protective review should flatten that decline.
Baselines
The naive reference point is the static leaderboard as currently published. It consists of each target model's ALERT safety score under Llama Guard. ALERT reports, for example, 96.95% overall safety for GPT-3.5 and around 75% for Mistral. We recompute these scores on our timeless stratum to confirm we reproduce the setup, then report the same judge's scores on the post-cutoff stratum. The quantity of interest is the gap between the two, and whether it reorders models.
The strongest automated competitor is a frontier LLM judge used as LiveSecBench uses DeepSeek-V3.2, where it reports 98.5% agreement with human annotation. Its competitive strength is that it has the highest reported judge-human agreement in the file. We run it with the same prompts across all strata, give it the same token budget as our pipeline's judge, and tune its prompt on a held-out pre-cutoff split only. We add a retrieval-augmented version of this judge that receives the same news snippets our reviewers see. This isolates whether retrieval alone closes the temporal gap. We found no reported result on Lacuna for any safety judge's agreement with humans on post-cutoff harm items. The gap measurement is therefore new data rather than a replication.
The strongest human-protective competitor is whole-item distanced review: the same routed items shown intact, with HARMCHECK-style placeholders and a retrieved context panel, to one reviewer each. It uses the same reviewers, the same per-item time budget and the same routing. It is the direct ablation of our key ingredient, decomposition. A second ablation keeps decomposition but removes retrieval, which isolates the temporal-grounding part.
The upper bound on accuracy is full-exposure review by a separate subset of the gold panel. It is also the worst case on exposure. The closest reported number for effort reduction in hybrid annotation is DIALOCONAN's post-editing HTER of 0.175 to 0.448 (Human-Machine Collaboration Approaches). That work measured authoring effort, not harm exposure, so it motivates the design but is not a comparable metric.
Decomposed review beats the field if three conditions hold on the post-cutoff stratum. First, its agreement with gold is statistically indistinguishable from full-exposure review. Second, it cuts raw toxic tokens shown per decision by at least half relative to whole-item distanced review. Third, it exceeds the retrieval-augmented LLM judge's agreement by a margin larger than that judge's gap from gold on the timeless stratum.
Why not just…?
Why not just use a strong LLM judge, given LiveSecBench's 98.5% human agreement? This is the main threat to the first claim, and the file contains evidence pointing both ways. The 98.5% figure was measured on LiveSecBench's own items, which were built from human-verified seeds and are likely close to the judge's training distribution. Felkner et al.'s $R^2 = 0.14$ came from a harder, community-grounded setting. The stratified set settles the disagreement directly. If the frontier judge holds near-human agreement on post-cutoff items, the staleness hypothesis is wrong. The practical conclusion would then be to use the judge, and we would report that.
Why not just give the judge retrieval? Retrieval might fix staleness without any humans. RealTime QA found that 34% of open-book errors were retrieval failures, and that models fell back on outdated beliefs when retrieval missed. That suggests retrieval will leave a residue, especially for coded terms that news does not define. Coded language may also be exactly the case that needs lived experience. The retrieval-augmented judge baseline measures that residue. If it is near zero, human routing is unnecessary for temporal drift, and the contribution reduces to the drift measurement itself.
Why not just apply HARMCHECK-style masking to the whole item? This is a serious threat to the second claim, because masking already removes much of the raw exposure and keeps full context. The whole-item distanced baseline exists to test it. We expect decomposition to win on exposure, since no reviewer sees the full harmful narrative. We expect whole-item masking to win on compositional items. If whole-item masking matches decomposition on exposure and distress, we drop decomposition and recommend masking plus routing.
Why not just keep safety test sets private? Privacy slows contamination but not staleness. As the file notes for GSM1k and LiveSecBench, a withheld static set still ages as language and events move. A private set also does nothing about the judge's own cutoff.
Decisive Checks
The staleness claim stands or falls on one comparison. We compare each judge's agreement with gold across the three matched strata. We also compute the Spearman correlation between judge-scored and gold-scored rankings of roughly a dozen open and closed target models on each stratum. A drop in both agreement and rank correlation on the post-cutoff stratum confirms the claim, and no drop refutes it.
The review claim stands or falls on the three-way comparison among decomposed, whole-item distanced, and full-exposure review. That comparison reports agreement, exposure, distress change and within-session fatigue slope, with items labeled in advance by whether their harm is compositional. The most informative result would be a clean split. Decomposition would match full-exposure accuracy on separable items at far lower exposure and degrade on compositional ones. Routing could then use a compositional-harm flag to decide which items get decomposed and which get masked whole review.
References
- A Careful Examination of Large Language Model Performance on Grade School Arithmetic
- GPT is Not an Annotator: The Necessity of Human Annotation in Fairness Benchmark Construction
- Responsible Crowdsourcing for Responsible Generative AI: Engaging Crowds in AI Auditing and Evaluation
- Handling and Presenting Harmful Text in NLP Research
- Can Machines Help Us Answering Question 16 in Datasheets, and In Turn Reflecting on Inappropriate Content?
- ALERT: A Comprehensive Benchmark for Assessing Large Language Models' Safety through Red Teaming
- MART: Improving LLM Safety with Multi-round Automatic Red-Teaming
- Automatic LLM Red Teaming
- LiveSecBench: A Dynamic and Culturally-Relevant AI Safety Benchmark for LLMs in Chinese Context
- When Splitting Makes Stronger: A Theoretical and Empirical Analysis of Divide-and-Conquer Prompting in LLMs
- ASTRA: Autonomous Spatial-Temporal Red-teaming for AI Software Assistants
- RealTime QA: What's the Answer Right Now?
- Mitigating Temporal Misalignment by Discarding Outdated Facts
- Human-Machine Collaboration Approaches to Build a Dialogue Dataset for Hate Speech Countering