When a language model solves a math problem with reinforcement learning, the reward says only whether the final answer was right. It never says which claim along the way made the difference. We propose to have the model keep, at a few checkpoints in its chain of thought, a short written ledger of what it currently knows: facts it has established, assumptions it is relying on, and sub-goals still open. A separate exploration process then branches from those checkpoints by editing exactly one ledger entry, for example by retracting an assumption, negating a claimed fact, or forcing a case split. The policy is trained to prefer the continuation whose ledger led to success over its one-entry-different sibling. The learning signal becomes "this particular belief was the mistake." Our bet is that this kind of signal transfers to code and science better than token-level signals do, because tracking what is known and what is assumed is the same skill in every domain.
This links four research lines that have so far met only in pairs. Epistemic state lifting means replacing raw history with a summary of what the agent knows, as in subset constructions for imperfect-information games. Logical contrastive learning means training against hard negative reasoning steps. Decoupled exploration means a separate process that gathers data while the main policy is optimized. The fourth line, cross-domain transfer, asks why math-trained reasoning helps elsewhere. In our design the ledger is the lifted state, edits to it are the exploration actions, ledger-differing sibling pairs supply the contrast, and transfer to coding and expert science is the test of whether the whole thing matters.
Core Claim
Step-level contrast already helps reasoning. Chain of Preference Optimization pairs a winning Tree-of-Thought step against its pruned siblings (CPO), and EDIT shows that only about 4.7% of chain-of-thought tokens are the pivots that decide outcomes (Capture the Key in Reasoning). Our claim is narrower. Where to branch and what to contrast should be chosen in the space of knowledge states rather than the space of token prefixes. Contrasts that differ in exactly one belief should then produce more cross-domain transfer than contrasts at random or search-selected step boundaries, when compute is matched. If that claim fails, and pairs that differ at an arbitrary step transfer just as well, the ledger is decoration, and the proposal should be dropped.
Prior Work and Where This Sits
Three papers motivate the lifted-state view for language models. Beyond Markovian (BARL) treats reasoning as a Bayes-adaptive process with a posterior over candidate hypotheses, using reflection as information-seeking. It keeps that posterior implicitly across multiple rollouts and evaluates only on math benchmarks. Reducing Belief Deviation shows that agents whose internal beliefs drift into "belief-trap regions" suffer advantage inversion, meaning uninformative trajectory tails push the gradient the wrong way. Its fix, called T³, truncates those tails and discards them. We take the opposite stance: the point where a ledger stops changing is exactly where to branch, and the discarded tail becomes a hard negative. The classical result that exact knowledge-set lifting is EXPTIME-complete (Omega-Regular Games with Imperfect Information) constrains our design. The ledger is deliberately a small, lossy, model-written summary, not an exact belief set. The question is whether it is good enough to localize credit, not whether it is sound.
On the contrastive side, Iterative RPO supplies a technique we reuse. Adding a negative log-likelihood term on the preferred chain stops absolute probabilities from collapsing, which the authors call the "DPO sink." The same paper warns that whole-path preferences plateau after three to four rounds. CPO's analysis of longest-common-prefix gradient cancellation is the reason our pairs diverge at a single point. When two paths share a long prefix, the gradients on that shared part cancel, so the useful signal sits only at the divergence. Our pairs share everything up to one ledger edit by construction. CPO builds its pairs with offline Tree-of-Thought search; we build ours on-policy during RL.
On exploration, EEPO splits rollouts into two stages. When entropy collapses, it temporarily unlearns the first-stage modes on the sampling worker so the second stage explores elsewhere. Its unlearning acts uniformly on high-probability tokens, its threshold is fragile, and it is evaluated only on math. Our explorer is also decoupled from the main policy, but it acts on beliefs rather than token probabilities. SePoT is the closest structural precedent: it reconstructs hidden states consistent with public history and separates search from training, but it works only on symbolic game trees.
On transfer, CPL is the strongest prior evidence that abstraction above the token level carries over. It runs tree search over high-level plan steps and trains with step-level advantage preferences, so a math-trained DeepSeekMath-7B improves on HumanEval and GPQA. Reasoning Vectors shows that the weight difference between a GRPO-trained model and its supervised starting point carries transferable reasoning. Subtracting that difference costs 11.8% on GSM8K. The work of Shen et al. shows that reflection and backtracking learned from math RL reach logic and code. These papers say transfer is real. They do not say which training signal causes it, and that is the question we isolate. CPL's plans describe what to do next; our ledger records what is believed so far. The comparison between the two is central.
Research Program
We build a GRPO-style training loop with three parts.
The first part is the policy itself. It is lightly instructed, and briefly fine-tuned on a few thousand self-generated examples, to emit a short tagged ledger every few reasoning steps. A ledger holds at most about eight entries, each one line, and each typed as established, assumed, or open.
The second part is the explorer. It is a frozen copy of the policy, refreshed every few hundred steps and given a ledger-editing prompt. At a checkpoint it proposes edits that each change exactly one entry: retract, negate, or split into cases. Checkpoints are chosen either where the ledger has stopped changing over a window, which is the belief-trap signal from T³, or at random, and we compare the two. The explorer only generates data; it is never the policy we deploy.
The third part is credit assignment at the ledger level. For a ledger $L$ at a checkpoint and one entry $e$ in it, we estimate the value of that entry as
$$\Delta(e) = \hat V(L) - \hat V(L \text{ with } e \text{ edited}),$$
where $\hat V$ is the fraction of $k$ continuations from that ledger that reach a verified-correct answer. Pairs whose $|\Delta(e)|$ exceeds a margin become preference pairs that diverge exactly at the edited entry. They are trained with the DPO+NLL objective from Iterative RPO, alongside the ordinary GRPO outcome reward on full trajectories. The added cost is the branch rollouts. We match total generated tokens against every baseline, so ledger-branched training sees fewer full trajectories in exchange.
The analysis asks two things. First, on held-out problems, does a high-$\Delta$ entry correspond to a human-judged pivotal error more often than CPO-style sibling steps do? Second, does the trained model keep writing and using ledgers on code and science prompts without being told to?
Baselines
The naive reference points are the base DeepSeekMath-7B model, and the same model after the short ledger-format fine-tune with no RL. The second shows whether writing ledgers alone changes anything.
The main RL reference is plain GRPO on the same data, with the same token budget and the same hyperparameter search budget. This is the setting in which Reasoning Vectors and Shen et al. already observe transfer, so any gain must exceed it.
EEPO is the strongest decoupled-exploration competitor. We run it with its published two-stage procedure and tune its entropy threshold within our shared tuning budget. It tests whether breaking modes at the token level gives the same benefit as breaking them at the belief level. We found no reported transfer result for EEPO beyond math competitions on Lacuna, and no numeric MATH score for it in the file.
CPL with Step-APO is the strongest existing transfer method and the main competitor on our question. We use DeepSeekMath-7B specifically so the comparison is direct. The file reports CPL's in-domain gain as "+6.5% on MATH" for DeepSeekMath-7B. The excerpt available to us cuts off before its HumanEval and GPQA numbers, so we treat those as no reported result found on Lacuna. We will rerun CPL at our token budget rather than quote it. Beating CPL means a higher average score across HumanEval, GPQA, and BBH at matched compute, with in-domain MATH no worse than CPL's.
For context on contrastive training, Iterative RPO reports Llama-2-70B-Chat rising "from 12.5% to 20.8%" on MATH and "from 55.6% to 81.6%" on GSM8K. Those numbers come from a different base model and only frame scale; they are not a target.
Two ablations isolate the key ingredient. In the first, the branching and contrastive loss are identical but branch points and edits fall on random reasoning steps instead of ledger entries. This is on-policy CPO, and it is the decisive ablation. In the second, ledger branching is kept but the contrast targets a random entry rather than the high-$\Delta$ one. A further variant runs T³-style truncation at stagnant checkpoints instead of branching, to test whether reusing the discarded tails matters.
Why not just…?
Why not just run GRPO longer, since math RL already transfers? Shen et al. and Reasoning Vectors show transfer arises from plain RL. More GRPO steps at equal token budget is therefore a real threat. We report transfer score against in-domain MATH accuracy along the whole training run. If our curve lies on GRPO's, we are only moving faster along the same path. Only a higher transfer score at equal MATH accuracy counts as a win.
Why not just do step-level contrast on-policy without any ledger? This is the most serious alternative. CPO's argument against prefix cancellation applies to any single-point divergence, and EDIT suggests the pivots are few enough that random branching may still hit them sometimes. Ablation one tests this directly. If it matches the full method on transfer, we will conclude the ledger adds nothing and say so. Our prediction that it will not rests on one concrete difference: a random-step edit changes surface text, while a ledger edit changes a belief that has to stay consistent across later steps.
Why not just make the model plan first, as CPL does? CPL's evidence for plan-level transfer is strong, and its plans are already domain-agnostic. But plans cannot represent a false belief that the model holds, which is exactly the failure that belief-trap analysis identifies. A hybrid that branches on plan steps under our on-policy, token-matched loop is cheap to add. If it equals ledger branching, the contribution reduces to "on-policy CPL," which is still useful but much smaller.
Why not just truncate stuck trajectories? T³ cleans the training buffer but throws away information. The truncation variant shows whether turning those tails into negatives matters, using the same compute.
Proposed Experiments
Training uses EEPO's split of 8.5K MATH problems at levels 3 to 5 (MATH) on DeepSeekMath-7B. Every method gets the same total generated tokens, the same verifier, and the same number of tuning trials. Every result is averaged over three seeds.
In-domain evaluation uses the MATH test set, GSM8K, AMC, AIME, and OlympiadBench. Transfer evaluation uses HumanEval for code, GPQA for expert science, BBH for symbolic logic, and ARC-C. These are the benchmarks on which CPL, Reasoning Vectors, and EDIT made their transfer claims. We report pass@1 throughout. We also report the gap between transfer and in-domain accuracy, measured at matched in-domain accuracy.
The decisive check is the comparison between ledger branching and random-step branching on transfer at matched MATH accuracy. A gain of at least two points on the transfer average, outside seed variance, supports the claim; anything smaller rejects it.
Two supporting checks follow. In a test-time knockout, we suppress ledger tags while the model generates on HumanEval and GPQA. If transfer depends on the learned state-tracking, accuracy should drop for our model but not for GRPO. In a pivot-localization study, we hand-label 300 failed MATH solutions and measure how often the top-$\Delta$ entry contains the first real error, compared with the CPO sibling step at the same budget. If localization is no better than CPO's, the mechanism we propose is not what drives any transfer gain we observe.
References
- Chain of Preference Optimization: Improving Chain-of-Thought Reasoning in LLMs
- Capture the Key in Reasoning to Enhance CoT Distillation Generalization
- Beyond Markovian: Reflective Exploration via Bayes-Adaptive RL for LLM Reasoning
- Reducing Belief Deviation in Reinforcement Learning for Active Reasoning of LLM Agents
- Algorithms for Omega-Regular Games with Imperfect Information
- Iterative Reasoning Preference Optimization
- EEPO: Exploration-Enhanced Policy Optimization via Sample-Then-Forget
- Look-ahead Search on Top of Policy Networks in Imperfect Information Games
- CPL: Critical Plan Step Learning Boosts LLM Generalization in Reasoning Tasks
- Reasoning Vectors: Transferring Chain-of-Thought Capabilities via Task Arithmetic
- Satori: Reinforcement Learning with Chain-of-Action-Thought Enhances LLM Reasoning via Autoregressive Search
- Measuring Mathematical Problem Solving With the MATH Dataset