Contrastive Reasoning Alignment: Reinforcement Learning from Hidden Representations

Haozheng Luo, Yimin Wang, Jiahao Yu, Binghui Wang, Yan Chen

The Hidden Vulnerability of Large Reasoning Models

As Large Language Models evolve into Large Reasoning Models (LRMs) like DeepSeek-R1 and Qwen3-Thinking, they gain the ability to generate extensive "reasoning traces" or Chain-of-Thought processes before delivering a final answer. While this significantly boosts their problem-solving capabilities, it introduces a novel security vulnerability: Superficial Safety Alignment (SSA).

SSA occurs when a model generates a seemingly safe refusal to a malicious prompt (e.g., "I cannot help with that"), but its internal reasoning trace actively processes harmful, unethical, or leaked information. Because traditional safety training focuses almost exclusively on the final output, the internal "inner monologue" remains unaligned. This discrepancy makes LRMs highly susceptible to sophisticated jailbreak attacks that exploit unsafe internal logic to bypass output filters.

Introducing CRAFT: Aligning the Inner Monologue

To address this, researchers propose CRAFT (Contrastive Reasoning Alignment), a red-teaming alignment framework that shifts safety optimization from the output text down to the model's latent space—its hidden mathematical representations.

CRAFT ensures that a model's internal reasoning is as safe as its final response through a two-phase process:

1. Latent Contrastive Learning for Reasoning (LCLR)

First, CRAFT structures the model's hidden states. Using contrastive learning, it maps out a "safety manifold" where safe, unsafe, and "rethink" (transitional) reasoning traces are geometrically separated into distinct regions. This creates a clear spatial boundary between benign and harmful internal computations.

2. Reinforcement over Reasoning Latents (R2L)

Building on this structured latent space, the framework fine-tunes the model using Group Relative Policy Optimization (GRPO). The optimization relies on a tripartite reward function:

  • Latent Semantic Reward ($R_{ls}$): Pushes the model's hidden states toward the established "safe" region of the latent space.
  • Textual Safety Reward ($R_{txt}$): A standard verification step ensuring the final generated text is safe.
  • Latent-Textual Consistency Reward ($R_{cons}$): The core innovation of CRAFT. It heavily penalizes the model if its internal latent state indicates one safety level while its text output indicates another.

Theoretically, the authors prove that this consistency reward eliminates superficially aligned policies by ensuring they cannot exist as local optima during the training process.

Key Insights: The Geometry of Thought and the "Negative Alignment Tax"

The research reveals fascinating properties about how LRMs process safety. Safe and unsafe reasoning traces naturally occupy entirely different regions in a model's hidden state space. Interestingly, "rethink" traces—moments where the model catches a harmful thought and pivots to a safe refusal—act as a geometric bridge between these regions.

Furthermore, safety alignment typically comes with an "alignment tax," where a model's general reasoning capabilities degrade as it becomes safer. Surprisingly, CRAFT exhibits a negative alignment tax. By forcing high-quality, safety-aligned reasoning trajectories, the model receives better overall logical signals, actually improving general reasoning performance by an average of 8.0% on complex benchmarks.

Empirical Results

CRAFT was rigorously evaluated on two state-of-the-art reasoning models: Qwen3-4B-Thinking and R1-Distill-Llama-8B. The models were tested against safety benchmarks (StrongReject, JailbreakBench) and performance benchmarks (AIME 2024, Minerva, MATH-500, LiveCodeBench).

  • Safety Improvements: CRAFT delivered an average 79.0% improvement in reasoning safety and an 87.7% improvement in final-response safety over the base models.
  • Jailbreak Robustness: Under aggressive automated attacks like GPTFuzzer and AutoDAN, CRAFT maintained high safety rates, whereas base models experienced near-total safety collapse.
  • State-of-the-Art Performance: CRAFT consistently outperformed existing defense methods, such as IPO and SafeKey, in both safety metrics and general reasoning tasks.

Limitations and Future Directions

While highly effective, CRAFT has limitations. The reliance on GRPO makes the framework computationally expensive, though researchers note that higher compute budgets could yield even stronger alignment. Additionally, the "safety heads" and latent prototypes are specific to individual model families; a safety head trained on a Llama architecture cannot be directly transferred to a Qwen model due to differing internal geometries.

Finally, the framework has yet to be tested against adaptive "latent-aware" attacks—adversaries specifically designed to manipulate the hidden states CRAFT protects. This remains a critical frontier for future LRM safety research.

Summary updated

Abstract

We propose CRAFT, a red-teaming alignment framework that leverages model reasoning capabilities and hidden representations to improve robustness against jailbreak attacks. Unlike prior defenses that operate primarily at the output level, CRAFT aligns large reasoning models to generate safety-aware reasoning traces by explicitly optimizing objectives defined over the hidden state space. Methodologically, CRAFT integrates contrastive representation learning with reinforcement learning to separate safe and unsafe reasoning trajectories, yielding a latent-space geometry that supports robust, reasoning-level safety alignment. Theoretically, we show that incorporating latent–textual consistency into GRPO eliminates superficially aligned policies by ruling them out as local optima. Empirically, we evaluate CRAFT on multiple safety benchmarks using two strong reasoning models, Qwen3-4B-Thinking and R1-Distill-Llama-8B, where it consistently outperforms state-of-the-art defenses such as IPO and SafeKey. Notably, CRAFT delivers an average **79.0%** improvement in reasoning safety and **87.7%** improvement in final-response safety over the base models, demonstrating the effectiveness of hidden-space reasoning alignment.
BibTeX
@inproceedings{luo2026contrastive,
  author = {Haozheng Luo and Yimin Wang and Jiahao Yu and Binghui Wang and Yan Chen},
  title = {{Contrastive Reasoning Alignment: Reinforcement Learning from Hidden Representations}},
  booktitle = {International Conference on Machine Learning},
  year = {2026},
  url = {https://openreview.net/forum?id=56cQPBIg4R},
}
Figure