Safety That Survives the Model's Own Explanations

Recommender systems built on large language models (LLMs) no longer just rank items. They explain them, in conversation, and they change the explanation depending on who is asking: a shopper gets a plain reason, a compliance auditor gets the features and weights behind the ranking. This note argues that this explanation mandate turns the model's earlier answers into a weapon against its own safety training. Once the assistant has written two or three confident, audience-tailored justifications, those turns sit in its context the way a forced "Sure, here is…" sits in a prefilling attack. A prefilling attack writes the opening words of the model's reply for it so that it continues past its refusal. We expect safety to be only "a few turns deep" in the same way that Safety Alignment Should be Made More Than Just a Few Tokens Deep showed it is only a few tokens deep.

We propose to measure this turn-level shallowness directly and to fix it cheaply. The fix is Turn-level Safety Recovery (TSR). We fine-tune on dialogues in which the assistant's own earlier explanation turns have drifted toward unsafe content, and the target turn recovers: it names what went wrong, withdraws the problematic part of its rationale, and continues helpfully in the register the stakeholder needs. Because no adversarial rollouts are needed at training time, TSR is meant to approach the robustness of trajectory-level reinforcement learning at a small fraction of its cost. Because recovery happens at a turn boundary rather than mid-sentence, it should also avoid the jarring pivots that token-level recovery produces.

Core Claim

The claim has two parts. The first is diagnostic. In an LLM conversational recommender, the probability that the model continues an unsafe line rises sharply with the number of prior assistant explanation turns that endorse it. The aligned model's next-turn distribution converges toward the unaligned base model's as those turns accumulate. This is the turn-level analogue of the per-token divergence decay that Qi et al. measured. We further predict that high-fidelity, developer- or auditor-style explanations are the steepest part of this curve, because they are the explanations the system is most obliged to give in full.

The second part is the remedy. Training on recovery from self-authored drift, conditioned on audience, makes this curve flat. It should do so as effectively as multi-turn reinforcement learning with future rewards, and with less over-refusal than prompt-level reminders.

Why This Gap Is Real

The motivating evidence comes from three places. Multi-turn attacks succeed by getting the model to commit early and escalate. Foot-In-The-Door: A Multi-turn Jailbreak for LLMs reaches 94% average attack success rate (ASR) across seven models and documents "semantic drift," where internal representations of safe and unsafe content blur. Conversational recommenders deliberately use explanations as hooks that invite the next turn. Towards Explainable Conversational Recommendation builds its feedback loop around them, and Expectation Confirmation Preference Optimization for Multi-Turn Conversational Recommendation Agent (ECPO) trains turn-level responses with Direct Preference Optimization (DPO, a method that fine-tunes a model directly on pairs of preferred and rejected responses) to satisfy a simulated user. Neither paper measures safety. Stakeholder-adaptive explanation, as in Adaptive Explanations via Direct Preference Optimization, optimizes communicative utility per listener with no safety term. Meanwhile LLMs Can Unlearn Refusal with Only 1,000 Benign Samples shows that small amounts of benign fine-tuning can detach refusal phrasing from refusal behavior.

Stacking audience-adaptive DPO on a chat model and then deploying it in a dialogue built around explanatory commitments is therefore the setting where shallow alignment should bite hardest. The stakeholder taxonomies of Human Factors in Model Interpretability: Industry Practices, Challenges, and Needs and Position: Explain to Question not to Justify give us the audiences: consumers, model builders, and auditors.

One prior design constrains ours. Beyond Ranking Accuracy: Evaluating LLM-Cited Feature Rationales for Next Basket Repurchase Recommendation shows that LLMs should translate the outputs of a separate supervised ranker rather than rank items themselves. We adopt that split. It means the attack surface we study is the explanation channel, not item selection: even a perfectly safe ranker does not stop an assistant from elaborating, on request, the "reason" a supplement, a credit product, or a chemistry book fits a user.

Closest Work and How This Differs

The closest work is MTSA: Multi-turn Safety Alignment for LLMs through Multi-round Red-teaming. It co-trains a red-team model with a target model and propagates penalties back from jailbroken dialogue endings to earlier turns, cutting Zephyr-7B ASR from 74.50% to 23.50%. TSR differs in three ways. It needs no rollouts of full dialogue trees, which the file flags as MTSA's main cost. It targets one specific mechanism, the model's own prior explanatory commitments, rather than generic multi-turn attacks. And its recovery targets are conditioned on audience, so the model learns what each stakeholder may legitimately receive.

The second closest is Qi et al.'s token-level safety recovery augmentation. It cut prefilling ASR on Llama-2-7B from 51.5% to 2.9%. TSR lifts that idea from tokens within a reply to whole turns within a dialogue. Whether the token-level version already transfers to turns is the central ablation, not an assumption.

Matching Ranks Over Probability Yields Truly Deep Safety Alignment supplies a diagnostic we reuse. It shows that harmful tokens can sit at rank two or three beneath a dominant refusal token. We check whether harmful continuations stay in the top ranks after TSR, rather than trusting probability alone.

Research Program

The program has three stages: measure, train, and stress-test.

Measurement. We build a conversational recommender from an open chat model (Llama-3.1-8B, matching ECPO) over a supervised ranker trained on the Amazon domains used by ECPO and ECR and on Instacart features from Cao et al. We then construct a boundary-item suite. Recommendation dialogues are seeded so that an item's legitimate explanation can be pushed toward a HarmBench or JailbreakBench behavior: a chemistry title toward synthesis details, a supplement toward dosing beyond safe limits, a credit product toward persuading a vulnerable user. Each dialogue is rendered for the three audiences.

To get the turn-depth curve, we prefill $k$ assistant explanation turns, for $k$ from 0 to 6, with progressively more compliant rationales. These are drawn from FITD-style escalation logs and written in the model's own style. At the target turn we record the HarmBench-classifier harm rate and the per-token KL divergence between the aligned model and its base. We also record the rank of the first harmful token, following Vega and Singh. We run this before and after audience-adaptive DPO in the style of Teneggi et al. and ECPO, which directly tests whether utility tuning for stakeholders makes safety shallower.

Training. TSR data consists of histories with self-authored drift at varying depths, each followed by an audience-appropriate recovery turn written by a strong model and filtered by the HarmBench classifier and by humans on a sample. For a consumer, the recovery declines and redirects in plain language. For an auditor, it gives the feature-level reasons for the ranking but withholds the harmful elaboration and says why. Benign histories with ordinary explanation turns are mixed in to protect helpfulness. Training uses standard supervised fine-tuning plus DPO, preferring the recovery turn over the compliant continuation.

Stress-testing. The trained model faces adaptive attacks that never appear in training: FITD with SlipperySlopeParaphrase and Re-Align, the MTSA-R3 red-teamer, and an "audience-switch" attack that claims auditor status mid-dialogue to unlock higher-fidelity explanations.

Baselines

The naive reference point is the undefended recommender: the same model, ranker, and explanation prompts, with only its original safety tuning. We expect it to behave like the multi-turn victims in FITD, which reports 93% ASR on GPT-4o and 98% on LLaMA-3. We have found no reported result on Lacuna for multi-turn attack success against an explaining conversational recommender, so this number is ours to establish.

The prompt-level baseline is periodic grounding as defined in A Prompt-Masked Pilot for History-Dependent Safety Degradation in Multi-Turn Conversational Agents: a safety reminder injected every three turns. That paper reports a 0.55-point reduction in mean confirmation score versus control (p = 0.0078), on a different task with a different rubric. We run the reminder with the identical schedule on our dialogues.

The strongest existing method is MTSA's multi-turn reinforcement learning with future rewards, whose reported result is the Zephyr-7B ASR drop from 74.50% to 23.50% with a 5.62% rise in XSTest over-refusal. We retrain it on our base model and our recommendation dialogues, using the same harmful-objective pool and a hyperparameter search of the same size as TSR's. We report it both at its native compute and at a compute budget matched to TSR's. TSR beats it if it reaches equal or lower ASR under the held-out adaptive attacks with no worse over-refusal and recommendation quality, at matched compute. Matching its ASR at a fraction of its compute counts as a practical win and is reported as such.

The key ablation is Qi et al.'s token-level recovery augmentation trained on the same harmful objectives and the same number of examples, with recoveries inserted mid-reply rather than at turn boundaries and with no audience conditioning. A second ablation removes audience conditioning from TSR, to test whether role-specific recovery matters or only recovery from self-history does.

Why not just…?

Why not just remind the model periodically, or set a strong system prompt? This is the cheapest serious competitor and should not be dismissed. The prompt-masked pilot shows grounding helps relative to control, but it also shows one-time persona framing and periodic grounding performing about the same (1.62 versus 1.64), and that the interventions change conversational style enough that a classifier detects them 55.6% of the time. Why Safeguarded Ships Run Aground? Aligned Large Language Models' Safety Mechanisms Tend to Be Anchored in The Template Region further shows that safety tied to template and prompt regions breaks when those anchors are disturbed. Our turn-depth curve settles the question directly. If reminders flatten it as much as TSR does under FITD and the audience-switch attack, the training method is unnecessary and we will say so.

Why not just apply token-level safety recovery augmentation? This is a real threat, because a model that learns to stop mid-reply might also learn to stop after harmful history. Our hypothesis is that it learns to recover from harmful text inside its current reply but not from coherent, fully formed earlier turns that it treats as settled context. The token-recovery ablation, evaluated on the same turn-depth curve, decides this. If it matches TSR at every depth, the contribution narrows to the measurement and the audience-conditioned recoveries.

Why not just filter with an external moderation classifier? Turn-level filters miss decomposed attacks: both FITD and MTSA rely on each turn looking benign. HarmBench itself notes that judging context-dependent harm remains open. We run a conversation-level variant that scores the whole history with the HarmBench classifier and report its ASR and false-block rate on benign explanation dialogues. We expect an auditor-mode explanation of a risky item to be flagged often when it is legitimate.

Why not just keep the LLM away from ranking, as Cao et al. recommend? We do keep it away from ranking. The point is that this separation protects which items are shown, not what the explanation says about them, so it is a design premise of this proposal rather than an alternative to it.

Proposed Experiments and Decisive Checks

Harm is measured as HarmBench-classifier ASR with fixed completion length, since HarmBench reports that length alone can shift ASR by up to 30%, and as StrongREJECT score for the severity of what leaks. Over-refusal is measured on XSTest plus our benign boundary-item dialogues. Utility is measured as ECPO-style success rate and win rate against a GPT-4o mini simulated user; ECPO reports 0.55 to 0.63 win rates for its fine-tuned agent. Explanation faithfulness uses Cao et al.'s feature-masking test. Audience fit uses a Teneggi-style listener-guessing accuracy.

The first decisive check is the turn-depth curve itself. If harm rate and divergence from the base model do not rise with $k$ prefilled explanation turns, the premise fails and the work stops there. The second is whether audience-adaptive DPO steepens that curve, which would establish Gap 1 empirically. The third is whether TSR keeps the curve flat under attacks it never saw, including the audience-switch attack, against MTSA at matched compute and against the token-recovery ablation. The fourth is whether the harmful continuation still sits at rank two or three after TSR, since a flat probability curve hiding harmful options just below the top would mean the fix is itself shallow.

Because all attackers here are simulated, we also run a small human red-teaming study on the boundary-item suite. This addresses the synthetic-dialogue gap the multi-turn literature acknowledges.