Abstract: Deductive reasoning is the process of deriving conclusions strictly from the given premises, without relying on external knowledge. We define honesty in this setting as a model's ability to respond only when the conclusion is logically entailed by the premises, and to abstain otherwise. However, current language models often fail to reason honestly, producing unwarranted answers when the input is insufficient. To study this challenge, we formulate honest deductive reasoning as multi-step tasks where models must either derive the correct conclusion or abstain. We curate two datasets from graph structures, one for linear algebra and one for logical inference, and introduce unanswerable cases by randomly perturbing an edge in half of the instances. We find that prompting and existing training methods, including GRPO with or without supervised fine-tuning initialization, struggle on these tasks. In particular, GRPO optimize only for final task outcomes, leaving models vulnerable to collapse when negative rewards dominate early training. To address this, we propose ANCHOR, a reinforcement learning method that injects ground truth trajectories into rollouts, preventing early training collapse. Our results demonstrate that this method stabilizes learning and significantly improves the overall reasoning performance, underscoring the importance of training dynamics for enabling honest deductive reasoning in language models.
Introduction
Large Language Models (LLMs) are notorious "people pleasers." When presented with a set of premises, they often fabricate a conclusion even if the logic does not follow—a phenomenon we term a lack of deductive honesty. While previous research has explored "factual honesty" (knowing the limits of internal knowledge), this work focuses on the ability of models to recognize when a conclusion is or is not logically entailed by the provided prompt.
Standard training methods struggle to instill this behavior. Supervised Fine-Tuning (SFT) tends to overfit to demonstrations, while modern reinforcement learning (RL) methods like Group Relative Policy Optimization (GRPO) suffer from "early training collapse." If a reasoning task is sufficiently difficult, a model's initial exploration may yield zero correct answers, resulting in vanishing gradients and a failure to learn.
The ANCHOR Method
To overcome the limitations of outcome-based RL, we introduce ANCHOR (Augmented with Necessary Correct and HOnest Reasoning). In standard GRPO, a model generates multiple responses and is rewarded based on their relative quality. ANCHOR modifies this by deterministically injecting the ground-truth trajectory into every group of rollouts.
By including a "perfect" answer in the mix, ANCHOR ensures a persistent positive reference signal. This allows the model to contrast its failed attempts against the injected anchor, effectively adding an SFT-like gradient term to the RL update. As shown in Figure 4, this stabilizes training dynamics, maintaining healthy gradient norms and preventing the model from collapsing during the early stages of hard reasoning tasks.
Benchmarking Deductive Honesty
We evaluated our approach using two new benchmarks designed to isolate reasoning from factual recall:
- GRAPHLA (Linear Algebra): Systems of linear equations presented as natural language stories.
- GRAPHLI (Logical Inference): Problems based on propositional logic rules like Modus Ponens.
To test honesty, we randomly perturb or remove premises in half of the instances. In these cases, the only correct response is to abstain (e.g., outputting "Unknown").
Key Results
Our experiments on the Qwen model family (0.6B to 3B parameters) reveal that models do not naturally learn to abstain; without specific training, they treat "Unknown" as just another label rather than a logical state. However, ANCHOR enables a significant performance leap. On the GRAPHLA dataset, while standard GRPO performed near baseline, ANCHOR boosted accuracy to over 99% for the 1.7B model.
As illustrated in Figure 2, ANCHOR-trained models maintain higher accuracy even as task complexity (reasoning depth) and noise (irrelevant edges) increase. Most importantly, these models learned to output "Unknown" correctly on unanswerable queries, whereas baseline models almost always attempted to calculate a fabricated result.
Conclusion
Deductive honesty is a critical yet overlooked component of reliable AI. Our findings suggest that training dynamics, rather than just model scale, are key to enabling honest reasoning. While ANCHOR provides a robust safety net for training, future work should explore scaling these methods to larger architectures and more complex, non-synthetic reasoning environments.
Summary updated