Cheap safety alignment increasingly depends on machine-made labels. A small set of human judgments seeds an annotator, which can be a small model or a larger model judging its own outputs. That annotator then labels a large pool of prompts and responses, marking which reply is safer or better, and the target model is trained on these labels. The standard way to decide which machine labels to trust is the annotator's confidence. For general helpfulness data this works well. For safety data we expect it to fail in a specific way. Safety classifiers are most confident exactly where they are shallow: they refuse anything containing "kill" or "bomb," and they flag polite replies to hateful messages as unsafe because of the words nearby. Weighting training data by confidence therefore passes these keyword shortcuts on to the model being trained, in more concentrated form. The expected result is a model that over-refuses harmless requests and still misses harmful requests phrased in neutral language.
We propose to score each safety pseudo-label by whether it survives a counterfactual test, not by how confident the annotator is. For every labeled example, a language model writes two "twins." The first is a benign twin that keeps the alarming words but removes the harmful intent. The second is a harmful twin that keeps the intent but removes or disguises the alarming words. A label is trusted only if the annotator's preference flips the right way on both twins: it should prefer compliance on the benign twin and refusal on the harmful one. We then use this consistency score to weight a preference-optimization objective (training the model to favor the chosen response over the rejected one), starting from a few hundred human safety labels. We ask whether this gives safer and less over-refusing models than confidence weighting at the same labeling budget.
Core Claim
In low-resource safety preference learning, the errors that matter in machine-annotated data are correlated with surface lexical cues, not random. The annotator's confidence is therefore a poor, and possibly inverted, signal of label quality on exactly the examples that decide over-refusal. A cheap black-box check of counterfactual consistency, built with LLM-generated minimal pairs, identifies those errors. Weighting or filtering by it should improve two things together. It should reduce refusals of safe prompts on XSTest and EVOREFUSE-TEST, and it should keep or improve jailbreak resistance on StrongREJECT and HarmBench. The comparison is against confidence-weighted training on the same seed and pool.
The claim concerns the label-trust signal, not the data. The twins are used to judge labels. Whether they should also be added as training data is a separate question, and we test it as a competing hypothesis.
Where This Sits
Two lines of work supply the machinery. When Weak LLMs Speak with Confidence, Preference Alignment Gets Stronger introduces confidence-weighted preference optimization, CW-PO. A tiny annotator is trained on a 20% human seed, labels the rest of the pool, and each training pair is weighted by the annotator's confidence margin. The paper reports an average 5% gain in gold reward accuracy over training on 100% human labels, measured on general preference sets including HH-RLHF. Stackelberg Self-Annotation makes self-labeling robust by training against the worst-case label distribution near the observed one. From 2,000 UltraFeedback seed labels it reaches 24.44% length-controlled AlpacaEval 2.0 win rate on Mistral-7B, against 9.03% for plain DPO (direct preference optimization) on the same seed. Neither paper evaluates jailbreaks or over-refusal. Both treat label noise as something confidence or a robustness radius can absorb. We think safety label noise is systematic, so it falls outside both assumptions.
Three papers motivate the premise that safety labels fail along lexical lines:
- XSTest shows aligned models refusing safe prompts that contain sensitive words. Llama 2 with its default system prompt refused 38% of them.
- EVOREFUSE shows that safety tuning induces keyword-association shortcuts.
- Annotation alignment finds that LLM safety raters are "hate-stringent" and "advice-permissive." They flag neutral replies to toxic input as unsafe, and they pass dangerous medical or legal advice that contains no alarming words.
Both biases are what a surface-cue annotator would produce. Our two twin types are designed to catch them: benign twins catch the first bias and trigger-free harmful twins catch the second. That paper is purely diagnostic and proposes no correction. We turn its diagnosis into a training-time filter.
Other work constrains the design or competes with it. EVOREFUSE also generates pseudo-malicious prompts, and DPO on its 3,000 pairs cuts over-refusal of Llama-3.1-8B-Instruct by up to 45.96%. However, its search needs white-box logit access to the target model, and it produces training data, not a test of label quality. We use only black-box rewriting, and we compare against EVOREFUSE-ALIGN directly.
Safety-Tuned LLaMAs warns that adding safety data carelessly backfires. With 2,000 safety samples the model refused more than half of XSTest's safe prompts, so over-refusal must be a primary metric and not an afterthought. Do as I do (Safely) shows that LLM rewriting of safety examples into the task's own format works remarkably well. A 1% paraphrased safety mix brought attack success back to near 0%, where a 50% generic mix still left it at 64%. This tells us rewriting is a strong tool. It also warns that twins must match the pool's format, or the consistency test will measure format sensitivity instead of intent sensitivity.
ESSA evolves safety specifications with an LLM and trains on reasoning traces built from them. It uses an 8,000-example seed and a heavy search loop, and it does not ask which machine labels to trust.
Research Program
The setting is fixed across all methods. The seed is a human-labeled set of about 500 safety preference pairs, drawn from HH-RLHF harmlessness dialogues, plus an equal number of helpfulness pairs. The unlabeled pool has roughly 20,000 prompts: HH-RLHF harmless-base and red-teaming prompts mixed with general instructions. For each pool prompt, the base student produces two responses, one sampled with and one without a refusal-eliciting prefix, so that most pairs contrast compliance with refusal.
We use two annotators, reflecting the two regimes in the literature. The first is a weak annotator, a small model fine-tuned on the seed as in CW-PO. The second is a mid-size open instruction model used as a judge with an analyze-then-rate prompt, the setting where Annotation alignment found the strongest correlation with humans.
Twin generation is done by a separate open rewriting model with a fixed prompt. For each pool prompt $x$ it produces a benign twin $x^{+}$ and a harmful twin $x^{-}$:
- $x^{+}$ keeps the sensitive tokens of $x$ and replaces the intent with a clearly legitimate one, such as a fictional, historical, or safety-education context.
- $x^{-}$ keeps or strengthens the harmful intent while paraphrasing away every term on a trigger lexicon. The lexicon is built from the tokens most predictive of refusal in the seed.
Following the format lesson of Do as I do (Safely), the twins keep the syntactic form of the original. Each twin gets its own compliance/refusal response pair, generated the same way as for the original. We validate twins with a verifier model plus a human audit of a random 300 twins. We report the verifier's precision and discard twins that fail it.
The consistency score for a pseudo-labeled pair uses the annotator's probability $p(\cdot)$ of preferring the refusal response. On the benign twin this probability should be low, and on the harmful twin it should be high. We define
$$s(x) = \big(1 - p(x^{+})\big)\cdot p(x^{-}),$$
which is near 1 only when the annotator distinguishes intent from vocabulary. The weight applied to the pair's DPO loss is $w(x) = c(x)\, s(x)$, where $c(x)$ is the CW-PO confidence margin on the original pair. We also run a variant using $s(x)$ alone and a hard-filter variant that drops pairs with $s(x)$ below a threshold tuned on the seed. The student is trained with weighted DPO on the pool together with the seed.
The work has three parts, each answering its own question.
The first is a label-level study, run before any training. On a held-out slice of human-labeled safety pairs we measure, for each annotator, how label accuracy varies with confidence. We stratify by whether the prompt contains trigger-lexicon terms. We then ask whether $s(x)$ separates correct from incorrect labels better than $c(x)$ does. On DICES-350 we check whether low-consistency items coincide with the conversations where LLM ratings diverge from the human mean in the hate-stringent and advice-permissive directions.
The second part is student training, using a 7–8B base model to match the scale of most prior data-lean results. The third is a scaling sweep over seed sizes of 100, 500 and 2,000 pairs, to locate where counterfactual checking stops mattering.
Baselines
The naive reference points bracket the problem. The low end is DPO on the human seed alone, with no pool. The ceiling is DPO on the entire pool with the human HH-RLHF labels where they exist. Any machine-labeling method must be judged between these two. We also report the base model with a safety system prompt. XSTest documents why this is a poor operating point, but it costs nothing and belongs on the chart.
The strongest direct competitor is CW-PO from When Weak LLMs Speak with Confidence. It is the published method for weighting machine preference labels from a small seed, and our proposal changes only its weighting signal. We run it with the same annotator, seed, pool, student, optimizer, number of steps and hyperparameter search budget: eight configurations per method, selected on a held-out validation split built from seed-like data and never from the test benchmarks. The paper reports an average 5% gold reward accuracy improvement over full human labels on general preference sets. No reported result found on Lacuna for CW-PO on XSTest, StrongREJECT or HarmBench, so we establish its safety numbers ourselves. Beating it means two things at matched StrongREJECT score (within noise across three seeds): lower safe-prompt refusal on XSTest and on EVOREFUSE-TEST, and no worse AlpacaEval 2.0 win rate.
The second strong competitor is distributionally robust self-annotation, SSAPO from Stackelberg Self-Annotation. It targets error compounding in machine labels by a different route. Its reported 24.44% versus 9.03% AlpacaEval 2.0 gap on Mistral-7B is the best evidence that robust self-labeling helps from a small seed, but that result concerns helpfulness only. We run it on the identical pool, with the robustness radius tuned within the same eight-configuration budget.
The third competitor is data-level mitigation of over-refusal. We fine-tune on the seed plus EVOREFUSE-ALIGN pairs, whose paper reports up to 45.96% over-refusal reduction on Llama-3.1-8B-Instruct. We also add the twins themselves as extra training pairs under CW-PO weighting, without using them to score labels. Both ablations cost the same amount of twin generation as our method. The twin-augmentation run isolates the key ingredient. If it matches consistency weighting, the benefit comes from the extra data and not from checking the labels.
Why not just…?
Why not just trust the annotator's confidence, as CW-PO does? The premise of this proposal is that confidence and correctness come apart on trigger-word prompts. The label-level study tests this before any student is trained. If confident errors are not concentrated on trigger-lexicon items, and $s(x)$ adds nothing to $c(x)$ in separating correct from incorrect labels, then CW-PO is sufficient and we stop. The evidence that this will not happen is indirect: XSTest and EVOREFUSE show that safety-tuned models refuse by vocabulary. Whether the annotator inherits this is exactly what we will measure.
Why not just add the twins, or EVOREFUSE-ALIGN, as training data and skip the weighting? This is the most serious threat. Contrast data alone already cuts over-refusal substantially in EVOREFUSE. Augmentation fixes what the student sees but leaves the wrong labels on the original pool in place. If the annotator is hate-stringent on 10% of the pool, the student is still pushed toward those errors, and the augmentation has to fight them. Our prediction is that augmentation recovers much of the XSTest gain but less of the advice-permissive side, meaning harmful requests with no alarming words. Weighting should help there because it removes the mislabeled permissive pairs. We report the equal-cost augmentation arm and the combined arm side by side. If augmentation alone matches us on every metric, we will say that the contribution reduces to a label-quality diagnostic rather than a training improvement.
Why not just use a frontier API model as the annotator? Cost is one reason, but the stronger reason is that frontier raters carry the same biases. Annotation alignment documents hate-stringent and advice-permissive errors in GPT-4, despite its r = 0.61 correlation with the human average. We include a large open judge, the best black-box-free stand-in available to us. If consistency weighting gives no gain on top of it, the lesson is that annotator scale removes the problem. That result would be publishable and would narrow our claim to weak and mid-size annotators.
Why not just use distributionally robust training? A worst-case radius around the observed labels protects against errors of bounded size. It does not tell the optimizer which direction the errors run, and keyword errors all run the same direction. The SSAPO arm tests whether robustness alone closes the gap.
Proposed Experiments and Decisive Checks
Evaluation uses benchmarks already in this literature:
- XSTest, reporting full-refusal rate on the 250 safe prompts and compliance on the 200 unsafe contrasts.
- EVOREFUSE-TEST, reporting refusal rate on its 582 pseudo-malicious prompts.
- StrongREJECT, whose rubric penalizes empty jailbreaks and whose automated grader correlates at about 0.90 with humans. For reference, ReSA reports a 0.9932 defense score on Llama-3.1-8B, obtained with a different method and a different data budget.
- HarmBench attack success rate, where ESSA reports 43.75% for base Qwen3-4B and 31.25% after its training.
- AlpacaEval 2.0 length-controlled win rate, plus MMLU and GSM8K, to track alignment tax.
At the annotator level we report accuracy against human labels on held-out HH-RLHF safety pairs, and agreement on DICES-350.
Leakage is the main validity risk. Twins could end up resembling XSTest templates, which would make the gains on it circular. We hold out all XSTest and EVOREFUSE-TEST prompts from the rewriter's prompt and examples. We report n-gram and embedding-nearest-neighbor overlap between twins and test prompts. EVOREFUSE-TEST serves as the out-of-generator check, since its prompts come from a different search process.
Four results would settle the question:
- The label-level test: $s(x)$ must beat $c(x)$ in separating correct from incorrect safety labels, with the gap concentrated on trigger-word prompts.
- At the student level, consistency weighting must lower XSTest and EVOREFUSE-TEST refusal relative to CW-PO without raising StrongREJECT scores. It must also beat the equal-cost twin-augmentation arm on at least the trigger-free harmful slice.
- The advantage should shrink as the seed grows from 100 to 2,000 pairs, since better seeds make better annotators. If instead it grows, the mechanism is something other than annotator noise and needs rethinking.
- Removing either twin type should selectively erase the corresponding gain. Dropping benign twins should hurt over-refusal metrics, and dropping harmful twins should hurt harmful-prompt compliance. This would confirm that each twin catches the bias it was designed for.
References
- When Weak LLMs Speak with Confidence, Preference Alignment Gets Stronger
- Stackelberg Self-Annotation: A Robust Approach to Data-Efficient LLM Alignment
- XSTest: A Test Suite for Identifying Exaggerated Safety Behaviours in Large Language Models
- EVOREFUSE: Evolutionary Prompt Optimization for Evaluation and Mitigation of LLM Over-Refusal to Pseudo-Malicious Instructions
- Annotation alignment: Comparing LLM and human annotations of conversational safety
- Safety-Tuned LLaMAs: Lessons From Improving the Safety of Large Language Models that Follow Instructions
- Do as I do (Safely): Mitigating Task-Specific Fine-tuning Risks in Large Language Models
- ESSA: Evolved Safety Specification Alignment