TL;DR
UpSafe°C converts a standard dense LLM into a modular, safety-aware sparse Mixture-of-Experts (MoE) by identifying and "upcycling" only safety-critical layers with dedicated safety experts. Paired with a two-stage training scheme and an inference-time "safety temperature" parameter ($\tau$), it enables dynamic, continuous control over the safety–utility trade-off on the fly without retraining.
What Problem Does This Paper Address?
Aligning large language models (LLMs) against jailbreaks and harmful generations usually requires painful compromises:
- External guardrails (like LlamaGuard) run independent classifiers, incurring latency and resource costs without using internal model representations.
- Inference guidance / safe decoding requires heavy test-time computation and often hurts instruction following.
- Standard post-training alignment (SFT/RLHF) provides a static safety boundary, can cause catastrophic forgetting on general utility benchmarks (e.g., coding, math), and lacks runtime controllability.
The authors address how to build an intrinsic, modular defense mechanism that maintains general utility, resists jailbreaks, and lets users dynamically dial safety up or down during inference.
Methods
UpSafe°C introduces a three-step pipeline:
[Dense Pretrained LLM]
│
▼ 1. Layer Scan: Linear probing identifies top-k safety-sensitive layers
[Safety-Critical Layers]
│
▼ 2. Upcycling & Two-Stage SFT: Copy original MLP into M-1 safety experts
• Stage 1: Train router + safety experts on harmful prompts (freeze general MLP)
• Stage 2: Train router as "Soft Guardrail" on mixed benign/harmful prompts
│
▼ 3. Inference: Dynamic "Safety Temperature" (τ ∈ [0, 1])
[Controllable Runtime Model] (Adjusts logit bias Δ and softmax scaling T(τ))
- Intrinsic Safety-Critical Layer Scan: Using a linear probe trained on last-token hidden representations across layers, the method computes a Safety Sensitivity Score (SS-Score, the probe's validation loss). Intermediate layers with the lowest loss separate benign from harmful representations best and are chosen as safety-critical (typically the top-3 layers).
- Safety-Aware Upcycling: Selected layers are transformed into sparse MoE layers. The original MLP is kept as the general expert ($E_0$), while multiple replicas are added as safety experts ($E_1, \dots, E_{M-1}$).
- Stage 1 (Safety Expert Training): $E_0$ is masked ($S_0 = -\infty$). Only the router and safety experts are trained on harmful queries using next-token prediction loss combined with a load-balancing auxiliary loss ($\mathcal{L}_{\text{AUX}}$).
- Stage 2 (Soft Guardrail Training): All experts are frozen. The router is trained on both harmful and benign prompts with a soft-guardrail classification loss ($\mathcal{L}_{\text{SG}}$) so it learns to route benign tokens to the general expert and harmful ones to safety experts.
- Inference-Time Safety Temperature ($\tau \in [0, 1]$): At inference, a continuous hyperparameter $\tau$ modulates routing logits $R$: $$\Delta_0 = (0.5 - \tau) \cdot C, \quad \Delta_{i \ge 1} = (\tau - 0.5) \cdot \frac{C}{N-1}$$ Logits are scaled by an adaptive temperature function $T(\tau) = 1.5^{1-|2\tau-1|} - 1 + \delta$ before softmax. Low $\tau$ enforces general expert execution; high $\tau$ drives tokens into safety experts; $\tau \approx 0.5$ provides balanced, smooth transitions.
Datasets
- Training: STAR-1 (1,000 harmful queries with safety reasoning, and 915 benign queries). Stage 1 uses the 1,000 harmful queries; Stage 2 uses the combined 1,915 examples.
- Safety & Jailbreak Evaluation:
- Red-teaming & toxicity: JBB (JailbreakBench, 100 behaviors), StrongReject (313 prompts), WildChat (370 toxic dialogues).
- Jailbreak attacks: WildJailbreak (250 real-world jailbreak prompts), plus adaptive attacks PAIR and ReNeLLM.
- Exaggerated safety / over-refusal: XSTest (non-refusal rate).
- Utility Evaluation: MMLU (general knowledge), GSM8K / Math-500 / AIME2024 / AIME2025 (mathematics), and HumanEval (code generation).
Key Insights and Ideas
- Safety awareness is localized: Probing proves that safety-relevant discrimination is concentrated in a few intermediate transformer layers (e.g., layers 13, 17, 18 in LLaMA-3.1-8B), rather than uniformly across the network.
- Upcycling isolates safety behavior: By leaving the original MLP untouched and cloning it into dedicated safety sub-networks, the model maintains its base capabilities on general tasks while giving the router the role of an internal classifier ("soft guardrail").
- Two-stage optimization is crucial: Jointly fine-tuning the router and experts in a single stage produces inferior trade-offs. Specializing experts first, then calibrating routing second, creates sharp safety boundaries without degrading general reasoning.
- Inference control without retraining: Safety temperature $\tau$ acts like a continuous rheostat—shifting the decision boundary on the Pareto frontier dynamically depending on the risk tolerance of the application.
Key Results
- Substantial Safety Gains: On WildJailbreak, UpSafe°C improves safety rates over SFT-only models by 7.8% on average (e.g., jumping from 77.6% to 90.8% on LLaMA-3.1-8B-Instruct), and hits 100% on JBB and StrongReject.
- Preserved Utility: Unlike conventional safety fine-tuning (which drops LLaMA-3.1-8B's MMLU/HumanEval/Math-500 average down to 53.14%), UpSafe°C maintains 60.39% (vanilla baseline is 60.21%).
- Low Overhead: For LLaMA-3.1-8B, upcycling only 3 layers increases total parameters by 1.59B, but active parameters at runtime only grow from 8.03B to 8.56B with a modest 7% FLOP increase (2.07T vs. 1.93T).
- Robustness Under Harmful Downstream Tuning: When fine-tuned on a poison-injected math dataset (GSM8K-PureBad), UpSafe°C dropped only 14 points on WildJailbreak (from 90.8% to 76.8%), whereas standard SFT collapsed by 44 points (77.6% down to 33.6%).
Limitations and Future Work
- Static layer scan reliance: For distilled reasoning models (like DeepSeek-R1-Distill), layer probing on the distilled checkpoint was noisy, requiring the authors to borrow safety-critical layer indices from the parent instruction-tuned model.
- Parameter storage overhead: Although active parameter count and FLOPs remain modest due to sparsity, duplicating MLPs in 3 layers still adds ~1.6B parameters to the disk and GPU memory footprint.
- Evaluation scale: The framework is evaluated up to 14B parameter models; validation on frontier-scale architectures (70B+) remains to be explored.
Summary updated