From Shallow to Deep: Pinning Semantic Intent via Causal GRPO

Shuyi Zhou, Zeen Song, Wenwen Qiang, Jiyan Sun, Yao Zhou, Yinlong Liu, Wei MaarXiv 2026 · arXiv

TL;DR

Current safety alignment in Large Language Models (LLMs) is often "skin-deep," leaving them highly vulnerable to adversarial prefix attacks like forcing the model to start with "Sure, here is." This paper introduces Two-Stage Causal-GRPO (TSC-GRPO), a framework that uses causal representation learning to "pin" the model's internal awareness of harmful intent, enabling robust late-stage refusals without sacrificing general model utility.

What Problem Does This Paper Address?

Despite extensive safety training via Supervised Fine-Tuning (SFT) and Reinforcement Learning from Human Feedback (RLHF), LLMs remain notoriously fragile against jailbreak attacks. A simple adversarial maneuver—such as prefix injection (e.g., forcing the model to begin its response with "Sure, here is")—can completely bypass safety guardrails, causing the model to generate prohibited content.

The authors diagnose the root cause of this vulnerability as Semantic Representation Decay. Through empirical analysis, they show that while a safety-aligned model initially recognizes the malicious intent of a query, this internal signal is highly unstable. As the model auto-regressively generates a forced compliant prefix, the internal "intent" representation is overwritten by the "style" of compliance. The model literally loses sight of the harm, turning safety alignment into a fragile game of keyword whack-a-mole.

Figure 1: Visualizing Semantic Collapse in Shallowly Aligned Models. (a) PCA projection shows that while harmful (red squares) and safe (blue circles) requests are distinct at t = 0, they collapse into a single, indistinguishable singularity after prefix injection (t > k); (b) The accuracy of a linear probe trained at t = 0 drops rapidly during the prefix injection phase (Phase 2) and remains slightly above the random chance level (0.5) during subsequent generation (Phase 3).

Methods

To transition from superficial behavioral patching to deep intervention, the authors propose TSC-GRPO, which operates in two coupled stages:

  1. Stage 1: Forging the Pin (Causal Intent Probe): Grounded in causal identifiability theory, the authors train a lightweight Multi-Layer Perceptron (MLP) probe ($g_\phi$) to disentangle invariant intent (Content, $c$) from stylistic perturbations (Style, $s$). By employing a novel hard-negative data augmentation strategy, the probe learns an alignment loss (minimizing projection distance between different style views of the same intent) and a uniformity loss (ensuring distinct intents are uniformly distributed on a hypersphere using the Kozachenko-Leonenko estimator). This creates a "Semantic Compass" that tracks malicious intent regardless of how it is wrapped.
  2. Stage 2: Pinning the Policy (Causal GRPO): The authors internalize this causal awareness into the LLM's policy using Group Relative Policy Optimization (GRPO). They construct "Fork-in-the-Road" training scenarios where the model is initialized with a harmful prefix and forced to choose between continuing the harm or pivoting to a refusal. Instead of sparse rewards, they introduce a cumulative token-by-token causal penalty. If the model continues the malicious narrative, the penalty grows; if it pivots to a safe transition, the penalty drops to zero.

Datasets

The framework utilizes several datasets for both training and evaluation:

  • Safety Training & Augmentation: Harmful queries are sampled from the HEx-PHI dataset, while safe queries are drawn from the Alpaca dataset.
  • Safety Evaluation: Fine-tuned models are rigorously tested against the AdvBench benchmark using a diverse suite of adversarial attacks (including GCG, AutoDAN, ReNeLLM, CodeChameleon, PAIR, DeepInception, Prefix Injection, and ICA).
  • Utility Evaluation: To ensure general capabilities are preserved, the models are evaluated on standard benchmarks: GSM8K (mathematical reasoning), HumanEval & MBPP (coding proficiency), and TruthfulQA (hallucination and truthfulness).

Key Insights and Ideas

  • Safety as a Representation Stability Problem: The core realization is that standard safety training only constrains the first few tokens of a response. True safety requires anchoring the safety signal at the representation level so it remains invariant throughout the entire generation trajectory.
  • Breaking Spurious Correlations ($c \perp s$): In standard text, polite or compliant phrasing is strongly correlated with safe content. By artificially pairing compliant prefixes with harmful queries during probe training, the authors break this spurious correlation, forcing the model to look deeper than surface-level lexical patterns.
  • The Power of Token-by-Token Penalties: Standard RLHF often suffers from sparse rewards. By penalizing the accumulation of harmful semantic signals token-by-token, the model learns a generalized defense policy: No matter how I started or what forced me, I must stop generating harm immediately.

Key Results

TSC-GRPO was validated across four prominent open-source base models (LLaMA-2-7B-Chat, LLaMA-3.1-8B-Instruct, Qwen2.5-7B-Instruct, and Qwen2.5-14B-Instruct).

  • Adversarial Robustness: TSC-GRPO consistently achieved the lowest Attack Success Rate (ASR) under almost every attack family. For instance, on the Qwen2.5-7B-Instruct model, it reduced the GCG attack success rate from 43.48% to just 1.35%, and completely zeroed out AutoDAN, Prefill, and ICA attacks.
  • Eliminating the "Alignment Tax": Unlike many safety interventions that degrade an LLM's intelligence, TSC-GRPO preserved or even slightly improved general utility. Models showed consistent improvements in coding proficiency on MBPP and maintained highly robust performance on GSM8K and HumanEval.

Limitations and Future Work

While the paper demonstrates remarkable defense capabilities, a few areas remain open for future exploration:

  • Dependence on Augmentation Quality: The causal probe's robustness inherently relies on the diversity and quality of the adversarial views generated in Stage 1.
  • Extension to Multi-Turn Conversations: The current framework primarily focuses on single-turn prefix and jailbreak vulnerabilities. Extending "Intent Pinning" to maintain safety across long, complex, multi-turn dialogues where adversarial context accumulates incrementally is a natural next step.

Summary updated

Abstract

Large Language Models remain vulnerable to adversarial prefix attacks (e.g., ``Sure, here is'') despite robust standard safety. We diagnose this vulnerability as Shallow Safety Alignment, stemming from a pathology we term semantic representation decay: as the model generates compliant prefixes, its internal malicious intent signal fades. To address this, we propose Two-Stage Causal-GRPO (TSC-GRPO), a framework designed to achieve intent pinning. First, grounded in causal identifiability theory, we train a causal intent probe to disentangle invariant intent from stylistic perturbations. Second, we internalize this causal awareness into the policy via Group Relative Policy Optimization. By employing a cumulative causal penalty within ``fork-in-the-road'' training scenarios, we force the model to learn that accumulating harmful tokens monotonically decreases reward, enabling robust late-stage refusals. Experiments show that TSC-GRPO significantly outperforms baselines in defending against jailbreak attacks while preserving general utility.
BibTeX
@misc{zhou2026from,
  author = {Shuyi Zhou and Zeen Song and Wenwen Qiang and Jiyan Sun and Yao Zhou and Yinlong Liu and Wei Ma},
  title = {{From Shallow to Deep: Pinning Semantic Intent via Causal GRPO}},
  year = {2026},
  eprint = {2603.02675},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2603.02675},
}

Loading related content...

Figure