Efficient Switchable Safety Control in LLMs via Magic-Token-Guided Co-Training

Jianfeng Si, Lin Sun, Zhewen Tan, Xiangzheng Zhang

TL;DR

This paper introduces a novel "magic-token-guided co-training" framework that allows a single Large Language Model (LLM) to dynamically switch between different safety behaviors (positive, negative, and rejective) at inference time. This approach achieves comparable safety performance to multi-stage methods while significantly reducing complexity and enabling fine-grained control over LLM outputs.

What Problem Does This Paper Address?

Large Language Models (LLMs) are increasingly deployed in user-facing applications, making content safety a critical concern. Existing alignment methods like Supervised Fine-Tuning (SFT) and Reinforcement Learning from Human Feedback (RLHF) often result in a single, fixed safety behavior. This inflexibility makes it difficult to adapt LLMs to diverse deployment scenarios (e.g., needing a more permissive mode for internal testing vs. a strictly safe mode for public use) without training separate models, which is costly and inefficient. Furthermore, these methods can be brittle, with models still susceptible to "jailbreak" prompts that elicit harmful content. The paper aims to address this by creating LLMs that are not only safe but also controllable and adaptable to various safety requirements.

Methods

The core innovation is a unified co-training framework that embeds three distinct safety behaviors into a single LLM during one SFT stage:

  • Positive (pos): Generates lawful, prosocial, and helpful responses.
  • Negative (neg): Generates unfiltered, risk-prone content, intended for internal red-teaming and adversarial testing.
  • Rejective (rej): Politely but firmly refuses unsafe requests.

These behaviors are activated at inference time using "magic tokens". These are not natural language prompts but rather random, system-level string identifiers embedded in the system prompt, making them stealthy and inaccessible to end-users. This allows for dynamic switching without requiring multi-stage training.

The training process uses multi-directional self-distillation. Instead of relying on external teacher models, the base LLM generates all three types of responses (pos, neg, rej) for a given prompt. This ensures that the learned behaviors are intrinsic to the model and not influenced by external biases. The training corpus is constructed by mixing these diverse responses.

A key concept introduced is the Safety Alignment Margin (SAM), a metric that quantifies the degree of behavioral separation in the model's output space, specifically by analyzing the first-token logits. A larger SAM indicates more distinct and controllable behavioral pathways.

Datasets

The paper utilizes both existing and custom datasets:

  • General Conversational Data (CHAT): Used to maintain fluency. Examples include the Llama-Nemotron project dataset (for English) and an in-house dataset (for Chinese).
  • Safety-Critical Data (SAFETY): Generated via multi-directional self-distillation. This includes prompts from:
  • English: NVIDIA Aegis 2.0, S-Eval (base and attack), XSTest, and HarmBench.
  • Chinese: In-house general risk and jailbreak attack evaluation datasets, and S-Eval (Chinese versions).

The paper also uses several evaluation datasets for benchmarking, including the aforementioned S-Eval, HarmBench, XSTest, and in-house datasets for both English and Chinese.

Key Insights and Ideas

  • Unified Safety Control: The framework demonstrates that multiple, distinct safety behaviors can be effectively learned and controlled within a single LLM, eliminating the need for specialized models for each safety scenario.
  • Stealthy and Controllable Switching: Magic tokens provide a robust and covert mechanism for switching LLM behavior at inference time, offering fine-grained control without exposing sensitive modes to users.
  • Intrinsic Safety Learning: Multi-directional self-distillation allows LLMs to learn safety behaviors from their own internal representations, reducing reliance on potentially biased external data.
  • Safety Alignment Margin (SAM): This novel metric provides a quantifiable and visualizable way to understand and validate the effectiveness of the co-training in creating distinct, separable behavioral pathways.
  • Cultural Adaptability: The framework can be extended to integrate region-specific safety policies (e.g., US vs. China), enabling culture-aware LLM alignment.

Key Results

  • The proposed model (MTC_en_pos) achieves a Constructive Safety Score of 97.55, matching the performance of two-stage SFT+DPO baselines (97.58).
  • The model demonstrates superior robustness to adversarial attacks, with only a 3.8% performance drop under attack, compared to an average of 21.5% for baselines.
  • The Safety Alignment Margin (SAM) for the MTC_en model is 0.131, an order of magnitude higher than baselines, indicating significantly better behavioral separation.
  • The magic tokens are robust to random generation and omission, defaulting to a safe mode, ensuring reliable fallback behavior.
  • The multi-policy model (MTC/MP_pos) achieves top scores in both English (97.45 Avg(en)) and Chinese (95.13 Avg(zh)) benchmarks.

Limitations and Future Work

  • Neg Mode Release: The "negative" (risk-exposing) mode is not publicly released due to its potential for misuse, although a safer variant (TinyR1-S-8B) is provided.
  • Future Work: The authors suggest exploring dynamic policy composition, mitigating potential misuse of the negative mode, and extending the magic-token control interface to other modalities and alignment dimensions. They also emphasize the shift towards switchable safety control as a fundamental design principle for future LLMs.

Summary updated

Abstract

Current methods for content safety in Large Language Models (LLMs), such as Supervised Fine-Tuning (SFT) and Reinforcement Learning from Human Feedback (RLHF), often rely on multi-stage training pipelines and lack fine-grained, post-deployment controllability. To address these limitations, we propose a unified co-training framework that efficiently integrates multiple safety behaviors: positive (lawful/prosocial), negative (unfiltered/risk-prone) and rejective (refusal-oriented/conservative) within a single SFT stage. Notably, each behavior is dynamically activated via a simple system-level instruction, or magic token, enabling stealthy and efficient behavioral switching at inference time. This flexibility supports diverse deployment scenarios, such as positive for safe user interaction, negative for internal red-teaming, and rejective for context-aware refusals triggered by upstream moderation signals. This co-training strategy induces a distinct Safety Alignment Margin in the output space, characterized by well-separated response distributions corresponding to each safety mode. The existence of this margin provides empirical evidence for the model's safety robustness and enables unprecedented fine-grained control. Experiments show that our method matches the safety alignment quality of SFT+DPO, with our 8B model notably surpassing DeepSeek-R1 (671B) in safety performance, while significantly reducing both training complexity and deployment costs. This work presents a scalable, efficient, and highly controllable solution for LLM content safety.
BibTeX
@inproceedings{si2026efficient,
  author = {Jianfeng Si and Lin Sun and Zhewen Tan and Xiangzheng Zhang},
  title = {{Efficient Switchable Safety Control in LLMs via Magic-Token-Guided Co-Training}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2026},
  url = {https://openreview.net/forum?id=R8QiGxRc5M},
}

Loading related content...

Figure