Where Does the Rejected Probability Go? Sibling-Set Preference Optimization for Step-Level Reasoning

Preference optimization methods teach a language model to favor one response over another. Most of them, including the widely used Direct Preference Optimization (DPO), constrain only the ratio between the probability of a preferred response and that of a rejected one. In math reasoning, the pairs used for this training are often near-misses: two versions of the same solution step that differ in a single number or operation. When training pushes the wrong step down, nothing in a pairwise loss says where that probability should go. It may move to the correct step. It may also move to wrong steps that resemble the rejected one but never appeared in the training data. We hypothesize that much of this freed probability leaks to those unseen errors. That leak would explain two puzzling findings: step-level preference training can raise contest-math accuracy while models still fail easy problems they "should" solve, and models still make basic computational slips.

We propose to measure this leak directly. We then replace the pairwise step loss with one that treats all candidate next steps under a shared solution prefix as a small closed distribution. The loss matches that distribution to a target that favors verified-correct steps, using a tunable divergence. It also adds a constraint that probability on correct steps may never fall below the starting model's level. The generalized-divergence literature supplies the method. Choosing the divergence decides whether probability concentrates on one correct step or spreads over several valid ones, and the "probability must not leave the chosen side" condition comes from work on displacement-resistant losses. The reasoning-limitations literature supplies the failure modes and the tests.

Core Claim

We make two claims. The first is a falsifiable diagnostic. After standard step-level DPO on near-miss pairs, held-out incorrect sibling steps (unseen alternative wrong continuations of the same prefix) gain probability relative to the reference model, while held-out correct siblings gain less. The second is a remedy. An offline objective that matches the policy to a target over the whole sibling set reduces this leak. That objective is an $f$-divergence, a family of distribution distances whose shape controls whether a model chases one answer or covers many, and it is paired with a floor on correct-step probability. Reducing the leak should improve accuracy on hard problems and consistency on easy ones, compared with pairwise losses of any divergence, with or without a likelihood anchor. If the diagnostic shows negligible leakage, the remedy has no reason to exist and we will report that negative result instead.

Relation to Prior Work

Three step-level methods motivate the problem and supply data pipelines. Step-DPO pairs the first wrong step with a correct continuation under a shared prefix and trains with standard DPO. Its finding that self-generated errors beat externally generated ones matters here, because self-generated near-misses are exactly where leakage to similar errors is plausible. SVPO uses Monte Carlo tree search to score sibling steps and builds pairwise step preferences plus a value head. Its search trees already contain the multi-sibling sets we need, but it trains pairwise and ties the value head to a reverse-KL implicit reward. PORT builds negatives by corrupting digits, which produces the purest near-miss pairs, again under standard DPO. Our method keeps their data sources and changes the objective from pairwise to set-level.

Two works constrain the design. Iterative RPO showed that DPO on reasoning lowers the likelihood of both winners and losers, and patched this with a likelihood term on the winner. Displacement-Resistant Extensions of DPO proved that a pairwise loss avoids this drop exactly when its generating function has its minimum at a ratio of at least one, and proposed SquaredPO. Both protect the chosen response. Neither says anything about where mass removed from the rejected response goes, which is the gap we target. Both are therefore baselines rather than straw men.

The divergence family comes from the generalized preference optimization work. $f$-DPO showed that non-reverse-KL constraints remain tractable offline and trade accuracy against entropy. $f$-PO unified offline methods as $f$-divergence minimization toward the optimal policy and introduced the tunable $\alpha$-divergence. $f$-DPG found Jensen-Shannon divergence a stable middle ground. All three were evaluated on chat, summarization, or sentiment and operated on whole responses, never on reasoning steps.

The closest work is DMPO. It matches a forward-KL target built from rewarded groups of sampled completions and showed that reverse-KL reinforcement learning collapses reasoning diversity and length. DMPO is online, works at the outcome level, and needs fresh verifiable rewards per batch. We are offline, work at the step level, reuse fixed sibling data, and add an explicit correct-mass floor that DMPO lacks.

On the evaluation side, ConsisEval measures whether models that solve a hard problem also solve its easy sub-problem. It names computational and off-by-one errors as dominant easy-problem failures, which are exactly the near-miss errors we predict leakage would amplify. Al-Khalili et al. show that final-answer accuracy hides unsound reasoning, so we cannot rely on accuracy alone.

Method

Fix a solution prefix $h$ (a partial derivation) and a set of $K$ candidate next steps $S(h)=\{s_1,\dots,s_K\}$ sampled from the reference model $\pi_{ref}$. Each step carries a correctness score $q_i\in[0,1]$, estimated from the fraction of rollouts from $h, s_i$ that reach the right answer, as in SVPO's tree search. Define the policy's renormalized distribution on the set, $\bar\pi_\theta(s_i\mid h)=\pi_\theta(s_i\mid h)/\sum_j \pi_\theta(s_j\mid h)$. Define the target $p(s_i\mid h)\propto \pi_{ref}(s_i\mid h)\exp(q_i/\beta)$, where $\beta$ sets how sharply the target prefers correct steps. Let $M^{+}_\theta(h)=\sum_{i:q_i>\tau}\pi_\theta(s_i\mid h)$ be the absolute probability the policy puts on steps judged correct, with $\tau$ a correctness threshold. The objective is

$$ \mathcal{L}(\theta)=\mathbb{E}_h\Big[D_{\alpha}\big(p(\cdot\mid h)\,\|\,\bar\pi_\theta(\cdot\mid h)\big)\Big]+\lambda\,\mathbb{E}_h\Big[\max\big(0,\ \log M^{+}_{ref}(h)-\log M^{+}_\theta(h)\big)\Big], $$

where $D_\alpha$ is the $\alpha$-divergence of $f$-PO and $\lambda$ weights the floor.

The first term fixes the shape of the distribution among siblings. Mass removed from a wrong step must reappear on the correct siblings in proportion to the target, rather than leaking outside the set. The second term is a set-level version of the displacement-resistance condition: absolute probability on correct steps may not fall below the starting model's level. Because the set is finite and explicitly enumerated, the normalizer is exact, and the tractability problems noted for $f$-DPO do not arise.

Varying $\alpha$ moves the method between mode-seeking and mode-covering. We expect mode-covering settings to preserve several valid derivations, which pass@k will show. With $K=2$, no floor, and $\alpha\to$ reverse KL, the objective is close to Step-DPO, which gives a clean ablation path.

Research Program

The work proceeds in three stages. The first stage measures leakage. Using Step-DPO-style first-error data and PORT-style digit corruptions on a 7–8B model, we hold out wrong and correct siblings for each training prefix. These include alternative digit corruptions and sampled wrong steps never seen in training. After standard step-level DPO, DPO plus likelihood anchor, and SquaredPO, we track how the log-probabilities of held-out siblings change. We also measure how much absolute probability the whole sampled sibling set retains.

The second stage trains the sibling-set objective across $\alpha$ values, $K\in\{2,4,8\}$, and with or without the floor. The third stage evaluates accuracy, consistency, diversity, and stability under extended training.

Baselines

The naive reference point is the instruction-tuned base model with no preference training, plus rejection-sampling fine-tuning: supervised training on only the correct sibling steps from the same data. This shows what positives alone buy.

The strongest existing pairwise step method is Step-DPO. Its reported MATH result is 70.8% for Qwen2-72B-Instruct. That figure is at 72B, and no reported result found on Lacuna for Step-DPO at 7–8B, so we rerun it ourselves.

For the anchored and displacement-resistant variants, we run Step-DPO's pairs with the Iterative RPO likelihood anchor and with SquaredPO. These are the most direct competitors, because they already fix the "chosen probability drops" pathology.

To isolate the set structure from mere pair count, we include an all-pairs baseline: pairwise DPO and SquaredPO over every correct×incorrect pair inside each sibling set. We also include pairwise $\alpha$-PO at matched $\alpha$, to separate the divergence choice from the set structure.

As external reference points at similar scale, the file reports SVPO at 67.2% on MATH with a 7B model (SVPO), and PCPO at 61.8% Pass@1 on MATH-500 with Mathstral-7B and 15.0% Pass@1 on AMC23 with Llama-3-8B-Instruct (PCPO). Their data and inference setups differ from ours, so we treat these as anchors, not head-to-head comparisons.

All in-house methods use the same prefixes, the same sibling samples (pairwise methods see subsets of the same sets), the same base checkpoint, the same number of optimizer steps, and an equal hyperparameter-search budget of 12 configurations each. For example, $\beta$ and the anchor weight $\gamma$ are tuned for the anchored baseline exactly as $\alpha$ and $\lambda$ are tuned for ours. Beating the baselines means two things. The method must show a statistically significant gain over the best tuned pairwise variant on MATH-500 and AMC23 across three seeds. It must also show a significant gain in ConsisEval consistency score, with no loss in pass@1.

Why not just…?

Why not just add the likelihood anchor, or use SquaredPO? These keep the chosen step's probability up, which addresses the drop documented by Iterative RPO and Pipano et al.. A pairwise loss still never constrains the many other steps, so mass pushed off a wrong step can land on its unseen near-neighbors. This alternative is a real threat. If the stage-one diagnostic shows that anchored or SquaredPO training does not raise held-out wrong-sibling probability, the leakage story is wrong for these losses, and our method should match them. We would report that outcome.

Why not just fine-tune on correct steps and skip negatives? Positives-only training cannot leak mass toward errors in the way we describe. It is also cheap. PORT reports that preference training beats supervised training on rationales, and Step-DPO's gains come specifically from contrasting self-generated errors. Still, rejection-sampling fine-tuning on the identical sibling data is in our baseline set. If it matches our method on consistency, that would show the negatives add little once leakage is controlled.

Why not just use more pairs per prefix? The all-pairs baseline tests this directly. If all-pairs DPO with a likelihood anchor closes the gap, the benefit came from data coverage rather than from normalizing over the set, and we would recommend the simpler method.

Why not just run online reinforcement learning with distribution matching, as DMPO does? DMPO requires sampling and verifying fresh groups every step, and it stalls when no sample in a group is correct. Our setting reuses a fixed offline tree of siblings. We do not claim to beat online training. The claim is that offline step-level training, which Step-DPO shows is very data-efficient, can be fixed cheaply.

Proposed Experiments and Decisive Checks

We train Llama-3-8B-Instruct and Mathstral-7B on step siblings built from GSM8K and MATH training problems, using both self-sampled first errors and digit corruptions. Evaluation uses MATH-500, GSM8K, AMC23, and OlympiadBench for accuracy. Pass@1 and pass@16 measure diversity of valid derivations. The ConsisEval math split's consistency score measures easy-given-hard reliability. For soundness, we run a program-of-thought variant scored by the AST code-structure judge of Al-Khalili et al.. That judge was itself only 81% accurate, so we use it as a secondary signal.

The first decisive check is the leakage measurement. It succeeds if pairwise DPO raises held-out wrong-sibling log-probability while our objective does not. The second is whether consistency-score gains track reduced leakage across all methods and seeds. If that correlation is absent, the mechanism claim fails even if accuracy improves. The third replicates the multi-epoch collapse test of Pipano et al. on MATH-500, training each method for four epochs. The fourth is an $\alpha$ sweep, which should show mode-covering settings trading a small pass@1 cost for higher pass@16. If even that trade does not appear, the divergence choice does not matter at the step level, and only the floor term is doing work.