Stabilizing Reinforcement Learning for Honesty Alignment in Language Models on Deductive Reasoning

Jiarui Liu, Kaustubh Dhole, Yingheng Wang, Haoyang Wen, Sarah Zhang, Haitao Mao, Gaotang Li, Neeraj Varshney, Jingguo Liu, Xiaoman Pan

TL;DR

Current language models struggle with "honest" deductive reasoning—knowing when to derive a valid conclusion versus abstaining when the provided premises are insufficient. When trained with reinforcement learning (RL) like Group Relative Policy Optimization (GRPO), models frequently collapse because early rollouts are uniformly wrong, causing relative advantages and gradients to vanish. This paper introduces ANCHOR, a technique that deterministically injects ground-truth trajectories into rollout groups, providing a guaranteed positive reference signal that prevents training collapse and significantly boosts reasoning accuracy and honest abstention.

What Problem Does This Paper Address?

Honesty in language models involves two dimensions: (1) knowing what the model internally doesn't know (factual uncertainty), and (2) recognizing whether an answer can logically be derived from the provided premises alone. Most research focuses on the former, leaving pure context-dependent deductive honesty underexplored.

When models face multi-step deductive problems, standard alignment methods fall short:

  • Supervised Fine-Tuning (SFT) memorizes specific demonstrations without learning to contrast correct steps against invalid deviations.
  • GRPO (Group Relative Policy Optimization) relies on relative reward differences among sampled candidate answers for a given prompt. On difficult tasks, early policy rollouts often fail entirely; when every rollout receives the same negative reward, advantage variance collapses to zero. Gradients vanish, halting learning or reinforcing trivial majority-class guessing (e.g., always predicting "Unknown").

Methods

The authors propose ANCHOR (Augmented with Necessary Correct and HOnest Reasoning) to bridge the gap between supervised demonstration and reinforcement exploration:

  • Ground-Truth Rollout Injection: In each training step, along with $n$ policy rollouts (e.g., $n=5$), the verifiable ground-truth reasoning trajectory is injected as an additional rollout ($n=6$).
  • Stabilized Gradient Dynamics: The authors mathematically prove that including this verified trajectory introduces an SFT-like gradient update term scaled by the positive advantage of the ground truth rollout, while preserving GRPO’s clipped importance weighting and group-relative credit assignment.
  • Exploration without Cold-Start Failure: Because every group contains at least one successful path, the relative advantage never collapses to zero, even on hard queries where the policy's self-generated rollouts all fail.

Datasets

To isolate pure deductive reasoning from factual recall, the authors formulate reasoning tasks over Directed Acyclic Hypergraphs (DAHs) and generate two balanced datasets featuring both answerable and unanswerable instances:

  1. GRAPHLA (Linear Algebra): Word problems structured around systems of linear equations (disguised as dish prices across restaurants). Unanswerable instances are generated by perturbing or deleting edges (equations) so that no path connects the known roots to the target variable. The target output is an integer or "Unknown".
  2. GRAPHLI (Logical Inference): Multi-step chains of propositional logic rules (Modus Ponens, Disjunctive Syllogism, Dilemmas, etc.) mapped to natural language event descriptions with distractor premises. Unanswerable cases are created via premise removal, false premise injection, or inverted conclusions. The target output is binary ("Yes" or "No").

Key Insights and Ideas

  • Untrained Models Deteriorate Rapidly with Depth: Across Qwen-2.5-3B-Instruct, Qwen-3-0.6B, and Qwen-3-1.7B, accuracy on answerable instances falls toward zero once the reasoning chain depth $k > 6$. On unanswerable queries, models fail to reliably abstain, revealing widespread "dishonest" overconfidence.
  • GRPO and SFT Succumb to Shortcut Hacking: On challenging settings, standard GRPO and SFT often collapse into predicting the majority class (e.g., guessing "Unknown" 100% of the time, resulting in near-zero accuracy on answerable items).
  • Curriculum Learning is Fragile: Easy-to-hard training helps in select instances, but is hyper-sensitive to difficulty calibration. For example, moving from $|V|=5$ to $|V|=8$ variables in the "easy" phase causes Qwen-2.5-3B to fail completely.
  • Process Rewards Fall Short: The authors extensively evaluated process reward models (PRMs)—including LLM-as-a-judge, step-level heuristics, and syntactic markers—and found them fragile, hard to tune, and overly domain-specific compared to ANCHOR’s simpler trajectory injection.

Key Results

Experiments across Qwen-2.5-3B, Qwen-3-0.6B, and Qwen-3-1.7B demonstrate substantial improvements:

  • GRAPHLA: Baseline GRPO and SFT plateau at ~50% overall accuracy (achieving ~100% on unanswerable by cheating, but 0% to 33% on answerable queries). ANCHOR raises Qwen-3-1.7B to 99.3% overall accuracy (99.3% on unanswerable and 99.3% on answerable). Combined with curriculum learning, ANCHOR elevates Qwen-2.5-3B from 50.0% to 98.7%.
  • GRAPHLI: On hard 15-step inference chains, SFT hovers near random guessing (~50%). ANCHOR brings Qwen-3-1.7B to 86.0% overall (and 92.3% when paired with curriculum learning).
  • Training Stability: Gradient norms and clipping fractions under ANCHOR decay smoothly and reliably, whereas vanilla GRPO displays erratic, noisy gradient spikes.

Limitations and Future Work

  • Reliance on Verifiable Ground-Truth Traces: ANCHOR requires access to full, correct reasoning trajectories during training, which are straightforward to generate in formal graph/math problems but harder to obtain for open-ended or ambiguous tasks.
  • Model Diversity: Empirical evaluation is concentrated on compact open-source models (Qwen variants from 0.6B to 3B parameters); scaling behaviors on larger models (>70B) remain to be explored.
  • Synthetic Deduction vs. World Knowledge: The tasks strictly decouple deductive reasoning from external real-world knowledge. Future research must examine how to reconcile context-restricted abstention with hybrid tasks requiring external factual recall.

Summary updated

Abstract

Reinforcement learning with verifiable rewards (RLVR) has recently emerged as a promising framework for aligning language models with complex reasoning objectives, but training stability remains a key challenge. Standard approaches often optimize only for final task outcomes, which can cause gradient instability when negative rewards dominate early learning. We study this issue in the context of deductive reasoning, a domain that isolates reasoning dynamics from external factual knowledge. To systematically analyze this behavior, we construct two graph-based reasoning datasets, one rooted in linear algebra and one in logical inference, each containing both solvable and unsolvable cases. We find that conventional optimization strategies, such as GRPO and curriculum learning, are sensitive to reward imbalance and task difficulty. To address these limitations, we propose ANCHOR, a reinforcement learning method that incorporates verifiable reference trajectories into rollouts to maintain stable optimization. This addition introduces a bounded positive reference signal that prevents gradient collapse. Experiments across multiple reasoning models show that ANCHOR improves convergence stability and consistency in multi-step reasoning. These results suggest a mathematically grounded approach to stabilizing reinforcement optimization in structured reasoning architectures.
BibTeX
@inproceedings{liu2026stabilizing,
  author = {Jiarui Liu and Kaustubh Dhole and Yingheng Wang and Haoyang Wen and Sarah Zhang and Haitao Mao and Gaotang Li and Neeraj Varshney and Jingguo Liu and Xiaoman Pan},
  title = {{Stabilizing Reinforcement Learning for Honesty Alignment in Language Models on Deductive Reasoning}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2026},
  url = {https://openreview.net/forum?id=XWgdXCLGAV},
}
Figure