TL;DR
Current language models struggle with "honest" deductive reasoning—knowing when to derive a valid conclusion versus abstaining when the provided premises are insufficient. When trained with reinforcement learning (RL) like Group Relative Policy Optimization (GRPO), models frequently collapse because early rollouts are uniformly wrong, causing relative advantages and gradients to vanish. This paper introduces ANCHOR, a technique that deterministically injects ground-truth trajectories into rollout groups, providing a guaranteed positive reference signal that prevents training collapse and significantly boosts reasoning accuracy and honest abstention.
What Problem Does This Paper Address?
Honesty in language models involves two dimensions: (1) knowing what the model internally doesn't know (factual uncertainty), and (2) recognizing whether an answer can logically be derived from the provided premises alone. Most research focuses on the former, leaving pure context-dependent deductive honesty underexplored.
When models face multi-step deductive problems, standard alignment methods fall short:
- Supervised Fine-Tuning (SFT) memorizes specific demonstrations without learning to contrast correct steps against invalid deviations.
- GRPO (Group Relative Policy Optimization) relies on relative reward differences among sampled candidate answers for a given prompt. On difficult tasks, early policy rollouts often fail entirely; when every rollout receives the same negative reward, advantage variance collapses to zero. Gradients vanish, halting learning or reinforcing trivial majority-class guessing (e.g., always predicting "Unknown").
Methods
The authors propose ANCHOR (Augmented with Necessary Correct and HOnest Reasoning) to bridge the gap between supervised demonstration and reinforcement exploration:
- Ground-Truth Rollout Injection: In each training step, along with $n$ policy rollouts (e.g., $n=5$), the verifiable ground-truth reasoning trajectory is injected as an additional rollout ($n=6$).
- Stabilized Gradient Dynamics: The authors mathematically prove that including this verified trajectory introduces an SFT-like gradient update term scaled by the positive advantage of the ground truth rollout, while preserving GRPO’s clipped importance weighting and group-relative credit assignment.
- Exploration without Cold-Start Failure: Because every group contains at least one successful path, the relative advantage never collapses to zero, even on hard queries where the policy's self-generated rollouts all fail.
Datasets
To isolate pure deductive reasoning from factual recall, the authors formulate reasoning tasks over Directed Acyclic Hypergraphs (DAHs) and generate two balanced datasets featuring both answerable and unanswerable instances:
- GRAPHLA (Linear Algebra): Word problems structured around systems of linear equations (disguised as dish prices across restaurants). Unanswerable instances are generated by perturbing or deleting edges (equations) so that no path connects the known roots to the target variable. The target output is an integer or
"Unknown". - GRAPHLI (Logical Inference): Multi-step chains of propositional logic rules (Modus Ponens, Disjunctive Syllogism, Dilemmas, etc.) mapped to natural language event descriptions with distractor premises. Unanswerable cases are created via premise removal, false premise injection, or inverted conclusions. The target output is binary (
"Yes"or"No").
Key Insights and Ideas
- Untrained Models Deteriorate Rapidly with Depth: Across Qwen-2.5-3B-Instruct, Qwen-3-0.6B, and Qwen-3-1.7B, accuracy on answerable instances falls toward zero once the reasoning chain depth $k > 6$. On unanswerable queries, models fail to reliably abstain, revealing widespread "dishonest" overconfidence.
- GRPO and SFT Succumb to Shortcut Hacking: On challenging settings, standard GRPO and SFT often collapse into predicting the majority class (e.g., guessing
"Unknown"100% of the time, resulting in near-zero accuracy on answerable items). - Curriculum Learning is Fragile: Easy-to-hard training helps in select instances, but is hyper-sensitive to difficulty calibration. For example, moving from $|V|=5$ to $|V|=8$ variables in the "easy" phase causes Qwen-2.5-3B to fail completely.
- Process Rewards Fall Short: The authors extensively evaluated process reward models (PRMs)—including LLM-as-a-judge, step-level heuristics, and syntactic markers—and found them fragile, hard to tune, and overly domain-specific compared to ANCHOR’s simpler trajectory injection.
Key Results
Experiments across Qwen-2.5-3B, Qwen-3-0.6B, and Qwen-3-1.7B demonstrate substantial improvements:
- GRAPHLA: Baseline GRPO and SFT plateau at ~50% overall accuracy (achieving ~100% on unanswerable by cheating, but 0% to 33% on answerable queries). ANCHOR raises Qwen-3-1.7B to 99.3% overall accuracy (99.3% on unanswerable and 99.3% on answerable). Combined with curriculum learning, ANCHOR elevates Qwen-2.5-3B from 50.0% to 98.7%.
- GRAPHLI: On hard 15-step inference chains, SFT hovers near random guessing (~50%). ANCHOR brings Qwen-3-1.7B to 86.0% overall (and 92.3% when paired with curriculum learning).
- Training Stability: Gradient norms and clipping fractions under ANCHOR decay smoothly and reliably, whereas vanilla GRPO displays erratic, noisy gradient spikes.
Limitations and Future Work
- Reliance on Verifiable Ground-Truth Traces: ANCHOR requires access to full, correct reasoning trajectories during training, which are straightforward to generate in formal graph/math problems but harder to obtain for open-ended or ambiguous tasks.
- Model Diversity: Empirical evaluation is concentrated on compact open-source models (Qwen variants from 0.6B to 3B parameters); scaling behaviors on larger models (>70B) remain to be explored.
- Synthetic Deduction vs. World Knowledge: The tasks strictly decouple deductive reasoning from external real-world knowledge. Future research must examine how to reconcile context-restricted abstention with hybrid tasks requiring external factual recall.
Summary updated