Chat models that refuse "how do I make [poison]" will often comply when the poison is named once in an innocent first turn and then referred to only as "it," "the stuff from earlier," or a trade name ten turns later. Fixing this requires training data full of such alias-masked conversations. That data is normally written and labeled by a larger "teacher" model, and the teacher has the same weakness. A teacher that cannot tell that "it" in turn eight means a nerve agent will write unconvincing attacks. Worse, it will mislabel which assistant replies were harmful. The student then inherits the blind spot.
We propose a fix for the labeling half of this problem, which we call label transport. Every training conversation is first written in explicit, canonical form, where the teacher's harm judgments are reliable. It is then rewritten into a masked surface form through tracked, meaning-preserving alias substitutions. Labels and target responses are carried over from the canonical version by construction. The teacher is never asked to judge text it cannot read. Each masked harmful dialogue is paired with a "benign twin" that has the same masking structure but a harmless referent, so the student learns to track what a pronoun points to rather than to fear pronouns.
Core Claim
In safety data generation, the teacher's reliability depends on surface form. It judges canonical harmful requests well and masked ones poorly. We claim three things. First, this gap can be measured directly. Second, it corrupts the labels of the standard "teacher writes and judges masked dialogues" pipeline. Third, transporting labels from canonical twins produces students that resist long-horizon alias masking on alias types never seen in training. This holds without the over-refusal that has accompanied every prior coreference defense. The new element is not alias-masked data as such. It is the decision to separate where the teacher judges (canonical text) from where the student is trained (masked text), and to make the safety signal depend on the referent through matched benign twins.
What Prior Work Establishes and Leaves Open
CoSafe motivates the proposal. It builds 1,400 three-turn dialogues from BeaverTails seeds in which the harmful keyword in the final turn is replaced by a pronoun or vague noun phrase. It reports 56% attack success on LLaMA2-Chat-7b and 13.9% on Mistral-7B. Its prompted defense, which asks the model to resolve coreferences first, made models refuse benign requests. CoSafe is evaluation-only, limited to three turns and simple pronouns, and uses GPT-4 to write its dialogues without asking whether GPT-4 itself misjudges masked text. We extend it to 5–15 turns and richer alias types: role descriptors, nicknames, euphemisms, and trade names. We also train against the masking rather than only measuring it.
Three teacher-driven alignment methods supply the training machinery we build on, and they define what we change.
- MTSA trains a red-team model to plan multi-turn goal decompositions. It aligns the defender with a "future reward" that penalizes early turns leading to a later jailbreak. Its attacks are template-driven and use explicit target keywords.
- MART runs iterative attacker–defender rounds, but each attack is a single-turn prompt.
- AdvChain has a teacher rewrite reasoning chains with deliberate temptations and corrections. It is single-turn, and its authors acknowledge it is bounded by the teacher's ability to write those drifts.
All three let the teacher both generate and judge in the same surface form. Our proposal removes that coupling.
Two further works shape the design.
- DIAG conditions a teacher on a student's failures to generate problems at the edge of its competence. In math, this raised effective sample yield from about 0.59 to 0.83 while static generation declined. We borrow this pacing idea but treat it as a secondary component that must earn its place in an ablation.
- Inside CORE-KG shows that type-aware, per-category coreference prompting cuts duplicate entity nodes in legal texts full of aliases like "the defendant" and "the driver." We reuse its type-aware alias inventory to structure our rewrites.
XSTest constrains the design. Llama 2 with its default system prompt refused 38% of safe prompts. Any defense that teaches suspicion of surface cues will reproduce this in multi-turn form, which is why benign twins are part of the core method rather than an add-on.
Research Program
Measuring the teacher's blind spot. We first measure this gap directly, before any training. We take canonical harmful dialogues and their tracked masked rewrites, and ask candidate teacher models two questions on each version: whether the final user turn seeks harmful content, and whether a given assistant reply is harmful. Because masked and canonical versions share a referent by construction, any disagreement is a surface-form error by the teacher. We report this gap by alias type and by the number of turns between where an entity is introduced and where it is used. If the gap is negligible for strong teachers, the main premise fails. The proposal then reduces to cheap augmentation, and we would say so.
Building the data. A teacher writes multi-turn conversations, with explicit harmful seeds from BeaverTails categories and a matched benign seed set. For each conversation the teacher produces the canonical form and safe target responses, and a separate judge validates those responses on the canonical text.
A rewriting stage then introduces each target entity explicitly in an early turn. Later references are replaced by an alias chain drawn from a type-aware inventory in the style of CORE-KG. A chain might run from the formal name, to a role descriptor, to a nickname or euphemism, to a pronoun. Each substitution is logged as a span-to-referent map.
A consistency check runs an independent coreference pass over the masked text. It discards rewrites where the logged referent is not recoverable from the conversation itself, which guards against masks a human could not resolve either. Labels, harmful-turn positions for MTSA-style future rewards, and safe target responses are copied from the canonical version.
Benign twins use identical chain structures over harmless referents, such as "it" meaning a Python process to kill. The student is trained to answer these helpfully.
Training. The student is trained with the same multi-turn preference objective as MTSA, including its future-reward credit assignment. Transported turn labels tell the objective which early turn set up the alias that a later violation exploits. Optional DIAG-style pacing adjusts two settings toward the student's observed failure boundary: alias obscurity and the gap between introduction and use.
Baselines
The naive reference point is the untouched instruction-tuned student: Zephyr-7B as in MTSA, plus LLaMA3-Instruct-8b and Mistral-7B-Instruct-v0.2 from CoSafe. This anchors attack success and refusal rates.
The strongest existing method is MTSA. It is the only cited approach that trains multi-turn safety with turn-level credit assignment. It reports Zephyr-7B attack success falling from 74.50% to 23.50% on AdvBench/HarmfulQA after three iterations, with a 5.62% increase in XSTest over-rejection. We rerun MTSA's pipeline with the same student, teacher, number of preference pairs, iterations, and hyperparameter search budget as our method. We do not rely on its published numbers alone, because those were not measured on masked dialogues. Beating MTSA means a clear reduction in attack success on our held-out long-horizon masked set and on CoSafe. It also means no larger rise in XSTest and masked-benign over-refusal, and no regression on AdvBench/HarmfulQA, MT-Bench, or AlpacaEval.
The key ablation is the standard pipeline over the same masked dialogues. The teacher judges and writes responses on the masked text directly, so data volume and surface forms are identical and only label provenance changes. If this matches label transport, our central claim is wrong.
A second ablation removes the benign twins and isolates their contribution to over-refusal. A third replaces DIAG-style pacing with uniform sampling over obscurity and distance.
We also include the CoSafe prompted coreference-resolution defense. No reported numeric result was found on Lacuna for its effect on attack success or helpfulness, so we will measure it.
Why not just…?
Why not just substitute aliases with rules, without any teacher? This is a real threat to the generation half of our pipeline, and we treat it as such. Rule-based substitution over canonical dialogues is itself a form of label transport, since it also tracks referents. We will run it as a variant: a gazetteer of nicknames and pronoun replacement, with labels carried over. If it matches teacher-written rewrites on held-out alias types, we will conclude that the teacher adds nothing at the rewriting stage and recommend the cheaper version. Our expectation is that it covers pronouns but not the long tail of euphemisms and contextual titles that Systematic Study of Long Tail Phenomena in Entity Linking shows static dictionaries miss. The claim about label provenance stands either way.
Why not just run a coreference resolver before the model sees each turn? We will run this pipeline baseline: resolve the masked conversation, then feed it to the unmodified student. For pronouns it may perform well. Two problems remain. Errors cascade, since a wrong antecedent early on poisons every later turn. And general-purpose extraction pipelines produce heavy noise; GraphRAG showed 30.53% duplicate nodes in the CORE-KG comparison. We expect the resolver to degrade on euphemisms and role descriptors, and to resolve benign twins toward dangerous readings. The held-out alias-type split will show whether that is true.
Why not just prompt the model to resolve references before answering? CoSafe tried this and found that models became over-cautious on benign requests. AdvChain shows that extra reasoning steps are themselves an attack surface, with about 54% attack success under reasoning hijack for standard safety-tuned models. We include this defense and score it on benign twins, where its over-refusal should appear directly.
Why not just classify the whole concatenated history with a safety classifier at each turn? The file reports attention dilution in long contexts (Jailbreaking in the Haystack) and the "contextual lull" in CoSafe. A history classifier trained on single-turn data is also judging masked text, so it inherits the same blind spot our first measurement quantifies. We run a classifier-over-history baseline and report its accuracy against distance from introduction to use.
Proposed Experiments
Evaluation sets. We evaluate on CoSafe (1,400 dialogues) and on a new held-out set of 5–15 turn alias-masked dialogues. Its alias types and harm categories are disjoint from training: for example, training on pronouns and role descriptors and testing on euphemisms and trade names. A separate attacker, built on the multi-turn jailbreak code in SpeakEasy, probes the trained models adaptively. Over-refusal is measured on XSTest and on held-out masked benign twins. Utility is measured on MT-Bench and AlpacaEval, and explicit-attack robustness on AdvBench and HarmfulQA.
Judging. Harm judgments use a judge from a different model family than the teacher, applied with the canonical expansion visible. A human-audited subset checks agreement, following CoSafe's practice, where agreement exceeded 85%.
Decisive checks. The project's claims stand or fall on four results.
- The teacher shows a measurable gap in judging canonical versus masked text. Without it, there is nothing to transport around.
- Label transport beats the same-data, teacher-labels-masked ablation on held-out alias types. This is the core claim.
- Benign twins keep masked-benign over-refusal at or below MTSA's level while attack success falls.
- The advantage grows with the distance between where an entity is introduced and where it is used. This is the long-horizon regime that CoSafe's three-turn setup could not reach.
References
- CoSafe: Evaluating Large Language Model Safety in Multi-Turn Dialogue Coreference
- MTSA: Multi-turn Safety Alignment for LLMs through Multi-round Red-teaming
- MART: Improving LLM Safety with Multi-round Automatic Red-Teaming
- AdvChain: Adversarial Chain-of-Thought Tuning for Robust Safety Alignment of Large Reasoning Models
- DIAG: Diagnostic Iterative Alignment and Generation for Data-Efficient Mathematical Preference Distillation
- Inside CORE-KG: Evaluating Structured Prompting and Coreference Resolution for Knowledge Graphs
- XSTest: A Test Suite for Identifying Exaggerated Safety Behaviours in Large Language Models
- Systematic Study of Long Tail Phenomena in Entity Linking
- Jailbreaking in the Haystack