A reasoning model trained with reinforcement learning on math or puzzles, where answers can be checked automatically, is often then used on tasks that have no checker: yes/no logical judgments, science multiple-choice, fact verification. The cheapest way to keep improving it there is self-training. You sample many solutions per question, treat the most common answer as a provisional correct label (a "pseudo-label"), and reward solutions that agree with it. This has a known failure mode from semi-supervised vision. If the model starts out biased toward one answer, say "Yes" or "Valid," the majority vote favors that answer on the questions it is unsure about. The reward then reinforces the bias, the next round of votes is more biased, and the model converges on a confident but systematically skewed answer distribution.
We propose replacing per-question majority voting with a batch-level assignment solved by optimal transport. Optimal transport is a way of matching items to categories at minimum total cost while forcing the category totals to hit target proportions. Here each question's vote mass is assigned to answer labels so that, across the batch, label frequencies roughly match an estimated prior. A reject bin lets questions with no clear agreement contribute no reward at all. The claim is narrow and testable. On target tasks where the transferred model has a measurable answer bias, transport-balanced pseudo-rewards should stop self-training from amplifying that bias, and the size of the gain should track the size of the initial bias.
Core Claim
Optimal transport pseudo-labeling was built to prevent exactly this collapse in vision. Towards Realistic Semi-supervised Learning assigns unlabeled images to classes with the Sinkhorn-Knopp algorithm under class priors. SoLar restricts the assignment to candidate label sets while tracking the class prior by moving average. P2OT adds a "virtual sink" that absorbs unconfident samples and relaxes the class totals to a soft penalty. None of these touches sequence generation or reinforcement learning on language models.
On the reasoning side, the transfer literature supplies the regime: CPL, Satori, 1-shot RLVR and Reasoning Vectors all show that verifiable math training moves scores on non-math benchmarks such as BigBench-Hard, ARC-Challenge and GPQA. That transfer stops at whatever the source training produced. Continuing to improve on the target needs a signal, and the label-free signals in the file have their own problems. MEDFACT-R1 uses pseudo-labels with no distribution constraint, plus a GPT-4 judge. JEPO and LaTRO still require the ground-truth answer string.
What is new here is using a constrained global assignment as the reward source for on-policy reinforcement learning on an unverified target domain. The work also makes the falsifiable prediction that the benefit is governed by measured answer bias. It is not a general claim that optimal transport "helps reasoning."
Method
For a batch of $N$ target questions drawn from one task with a closed label set of size $K$, we sample $G$ solutions per question and record $v_{ik}$, the fraction of samples for question $i$ that end in label $k$. The pseudo-label plan $P \in \mathbb{R}^{N \times (K+1)}$ solves
$$\min_{P \ge 0,\; P\mathbf{1} = \tfrac{1}{N}\mathbf{1}} \sum_{i,k} P_{ik} C_{ik} + \epsilon \sum_{i,k} P_{ik}\log P_{ik} + \tau\, \mathrm{KL}\big(P^{\top}\mathbf{1}_{1:K} \,\|\, (1-s)\,q\big).$$
Here $C_{ik} = -\log(v_{ik} + \delta)$ is the cost of assigning question $i$ to label $k$, with $\delta$ a small constant. Column $K+1$ is the reject bin with fixed cost $c_s$. The parameter $\epsilon$ controls how soft the assignment is. $\tau$ controls how strongly the label totals must match the prior $q$. $s$ is the fraction of mass allowed into the reject bin, and $P^{\top}\mathbf{1}_{1:K}$ is the vector of label totals over the first $K$ columns.
This is P2OT's unbalanced formulation transplanted to vote counts and solved by Sinkhorn iterations. The prior $q$ starts uniform and is updated by moving average of the plan's column sums, following SoLar. One change addresses P2OT's fixed schedule: the reject fraction $s$ is set each step from the batch's measured agreement rate, not from a sigmoid timer.
A sampled solution for question $i$ ending in label $k$ receives reward $N P_{ik}$. Questions whose mass lands mostly in the reject bin are dropped from the group-relative policy update (GRPO), so they contribute no gradient rather than a wrong one. The update stays on-policy. This matters because Does Math Reasoning Improve General LLM Capabilities? shows off-policy fine-tuning on teacher traces drops IFEval from 69.2 to 42.3, while on-policy RL keeps general ability near the base model.
Research Program
The study has three parts. First, a bias audit. Starting from source-trained checkpoints, we measure, per target task, the gap between the model's predicted label distribution and the true label distribution. Starting checkpoints are Qwen2.5-Math-1.5B after 1-shot or full-MATH RLVR, as in the 1-shot RLVR paper, and Qwen3-1.7B after multi-domain RL, as in TAC.
Second, self-training dynamics. We run label-free target-domain RL with each pseudo-reward rule and track three quantities over training: accuracy, the drift of predicted label frequencies, and the fraction of questions whose majority answer flips. Third, the robustness of the prior. We compare uniform, moving-average-estimated, deliberately misspecified, and oracle priors, because a misspecified prior is the known weakness of this family of methods.
Target tasks are chosen to vary bias on purpose. The main testbed is the closed-label subsets of BigBench-Hard: boolean, valid/invalid and yes/no tasks, where label bias cannot be removed by reordering options. ARC-Challenge and GPQA, run with option order randomly permuted, serve as controls. There, position bias is already removed, and we predict little or no gain from balancing.
Baselines
The naive reference point is the transferred source model with no target adaptation, evaluated greedily and with majority voting over the same $G$ samples. It tells us how much any self-training adds.
The strongest competitor is majority-vote self-reward with GRPO. Each sample is rewarded if it matches the question's plurality answer, with an agreement threshold below which the question is skipped. This is the direct per-question analogue of our method: same samples, same reward scale, same skipping logic, no batch constraint. No reported result was found on Lacuna for majority-vote self-reward RL on BigBench-Hard, ARC-Challenge or GPQA. For context on the size of transfer effects, the file reports +12.3% on BigBench-Hard for a 1.5B model from Reasoning Vectors, +8.6% on GPQA from CPL, and an average of 12.5% over base on ARC-Challenge zero-shot from LaTRO. LaTRO uses ground-truth answers, so it is an upper-reference rather than a label-free competitor.
We will run majority voting with the identical model, target questions, sample count, number of RL steps and hyperparameter sweep size as our method. Beating it means higher final target accuracy averaged over three seeds on the high-bias BigBench-Hard tasks, together with smaller label-frequency drift. Matching it on the permuted multiple-choice controls is the expected outcome, not a failure.
A second competitor is single-step prior correction. Each question's vote vector is divided by the batch's label frequencies before taking the majority, which amounts to one normalization pass of Sinkhorn. It gets the same tuning budget.
Two ablations isolate the key ingredients. One removes the reject bin, testing whether forcing every question into a real label hurts. The other replaces adaptive reject mass with P2OT's fixed sigmoid schedule. An oracle-prior run bounds what better prior estimation could buy.
Why not just…?
Why not just use majority vote as the pseudo-label? This is the real threat. On tasks where the starting model is nearly unbiased, majority voting and balanced assignment should produce almost the same labels, and we expect no difference there. The case for balancing rests on the confirmation-bias mechanism documented in vision. SoLar reports a 15.12% tail-class gain over the previous best on CIFAR10-LT. The Towards Realistic Semi-supervised Learning paper reports roughly 17% overall gain on CIFAR-100 with Sinkhorn assignment. Whether that transfers to language-model answer bias is an open question. The bias-versus-gain regression in our experiments settles it directly.
Why not just subtract the label prior once and skip the iterative solver? This may well capture most of the gain, and we treat that as a possible honest outcome. The difference is that a one-pass correction cannot jointly respect per-question votes and batch totals when votes are diffuse, and it has no reject bin. If the single-step baseline matches full Sinkhorn, the paper's conclusion becomes "prior correction is what matters." The reject-bin ablation would then show whether that piece contributes independently.
Why not just use an LLM judge or test-time voting instead of training? An LLM judge brings its own errors in exactly the expert domains of interest. GPT-4 scored 39% on GPQA at release (GPQA), and LaTRO notes that learned reward models invite reward hacking. Test-time voting is cheap to include, and we report it as the do-nothing baseline. It does not change the model, though, and Satori argues that internalized search outperforms external sampling. If voting at inference already matches our trained model's single-sample accuracy, the training cost is not justified, and we would report that.
Proposed Experiments and Decisive Checks
The decisive check is a per-task regression of the accuracy gain of balanced over majority-vote pseudo-rewards on the initial label bias measured in the audit. A positive slope with near-zero intercept supports the claim. A flat relationship refutes it.
The second check is dynamic. Under majority voting, predicted label frequencies on high-bias BigBench-Hard tasks should drift further toward the dominant label over training, and under transport they should not. We will plot both trajectories.
Third, prior robustness. With a deliberately wrong prior, for example 70/30 on a balanced binary task, the method should degrade gracefully toward majority-vote performance rather than below it. If it falls below, the reject bin and soft penalty are not doing their job.
Finally, to confirm that target adaptation does not erase other abilities, we report IFEval against the base score of 69.2 from the transferability audit, and MATH500 against the 73.6% 1-shot RLVR starting point, so that target gains are not bought by losing source skills.
References
- Towards Realistic Semi-supervised Learning
- SoLar: Sinkhorn Label Refinery for Imbalanced Partial-Label Learning
- P2OT: Progressive Partial Optimal Transport for Deep Imbalanced Clustering
- CPL: Critical Plan Step Learning Boosts LLM Generalization in Reasoning Tasks
- Satori: Reinforcement Learning with Chain-of-Action-Thought Enhances LLM Reasoning via Autoregressive Search
- Reinforcement Learning for Reasoning in Large Language Models with One Training Example
- Reasoning Vectors: Transferring Chain-of-Thought Capabilities via Task Arithmetic
- MEDFACT-R1: TOWARDS FACTUAL MEDICAL REASONING VIA PSEUDO-LABEL AUGMENTATION
- Beyond Verifiable Rewards: Scaling Reinforcement Learning in Language Models to Unverifiable Data
- Language Models are Hidden Reasoners: Unlocking Latent Reasoning Capabilities via Self-Rewarding
- Does Math Reasoning Improve General LLM Capabilities? Understanding Transferability of LLM Reasoning
- Transferability for General Reasoning: An Automated Curriculum for Multi-Domain LLM RL
- GPQA: A Graduate-Level Google-Proof Q&A Benchmark