Learning by Elimination: Self-Training Math-Trained Reasoners Where No Verifier Exists

Language models trained with reinforcement learning on math, where every answer can be checked automatically, often get better at code, science and logic too, even though they never trained on those. This is called cross-domain reasoning transfer. Once such a model reaches a domain with no automatic checker, such as first-order logic puzzles, expert science questions or commonsense yes/no questions, it stops improving: nothing tells it which of its own answers are right. The obvious fix is self-training. The model answers unlabeled target questions many times, treats the majority answer as a guessed label (a pseudo-label), and rewards itself for producing it. On hard questions the majority is often wrong. Training toward a wrong majority reinforces the mistake, a failure known as confirmation bias.

We propose to self-train on a weaker but safer signal: what the answer is not. A reasoning model asked "can this proposed answer be correct?" is often reliably able to reject a wrong option even when it cannot produce the right one. Learning from such exclusions is called complementary-label learning. We turn each confident rejection into a negative reward for any rollout that lands on the rejected answer. We weight this negative signal against any positive majority-vote signal using noise rates measured on the verifiable source domain, so no target labels are needed to set the balance. The claim is specific and falsifiable. On unverifiable target domains, rewards built from the model's own confident eliminations are more precise than rewards built from its own majority votes. That precision should turn into target-domain gains concentrated on the hardest questions, without the collapse that positive self-training shows.

Core Claim

Classic complementary-label methods show that "not class $k$" supervision is harder to get wrong than "is class $k$" supervision. In NLNL: Negative Learning for Noisy Labels, training on randomly drawn complementary labels resists 50% label noise on CIFAR-10 (91.17% vs 53.82% for cross-entropy). In In Defense of Pseudo-Labeling (UPS), adding low-uncertainty negative pseudo-labels cuts CIFAR-10 error from 22.02% to 8.18%. FixBi likewise uses low-confidence predictions as negative pseudo-labels to penalize unlikely classes. All three work on image classifiers with fixed label sets. None transfers directly to a reasoning model producing long answer traces.

NLNL's safety argument also constrains our design. A random negative collides with the true class with probability $1/(K-1)$, where $K$ is the number of classes. That is only safe for large $K$. Most unverifiable reasoning targets have small answer sets: FOLIO has three answers (true, false, unknown), GPQA and ARC-C have four options, and StrategyQA has two. Random negatives would be wrong a third to a half of the time. We therefore never use random or merely rare answers as negatives. Excluding rare answers would also suppress exactly the rare correct answers that reinforcement learning needs to amplify.

Instead, each exclusion needs independent evidence from a second "view" of the same model. The first view is generation: the model solves the question and produces an answer. The second is elimination: the model is shown the question and one candidate answer and asked to reason about whether that candidate is definitely wrong. This mirrors FixBi's idea of one model labeling data for another. Here the two roles are two prompts for one network, and only the rejecting role is trusted.

For an unlabeled target question $x$, let $e(a \mid x)$ be the fraction of $M$ sampled elimination judgments that reject candidate $a$. Let $E(x) = \{a : e(a\mid x) \ge \tau\}$ be the set of confidently eliminated answers, and let $\hat a_+(x)$ be the majority answer among generation rollouts when its vote margin exceeds a threshold. A training rollout $y$ with final answer $a(y)$ receives

$$r(y) = w_+\,\mathbb{1}[a(y) = \hat a_+(x)] \;-\; w_-\,\mathbb{1}[a(y) \in E(x)], \qquad w_\pm = \max(0,\, 2\pi_\pm - 1).$$

Here $\pi_+$ is the measured precision of the positive pseudo-label, meaning the probability that the confident majority answer is correct. $\pi_-$ is the measured precision of an elimination, meaning the probability that a rejected answer is truly wrong. The weight form follows from a simple fact. A reward that agrees with the true reward with probability $\pi$ and flips otherwise has expected value $(2\pi-1)$ times the true reward. Each signal is therefore scaled by how much real information it carries, and a signal no better than chance gets zero weight.

Both precisions are estimated on held-out verifiable math problems, bucketed by vote margin and rejection rate, and then carried over to the target. This is our answer to the multi-objective balancing problem. Domain Separation Networks documents composite losses whose fixed weights fail from one domain shift to the next. Unearthing Gems from Stones (BCPG-NSA) shows a mining coefficient that helps at 0.25 and hurts at −1.0. In both cases the weights were tuned by hand or against labeled data. Our weights come from measured noise rates. Whether those rates transfer from math to logic or science is an empirical question that the experiments test directly rather than assume.

Relation to Closest Work

Two recent papers also refuse to throw away failed reasoning traces, and they are the closest prior work.

ResRL projects tokens of wrong responses onto a subspace built from correct responses, so that valid intermediate steps escape the penalty. It reports a CodeForces rating of 1469.5 vs 1340.9. BCPG-NSA splits failed traces into segments and uses Claude-3.7-Sonnet plus a process reward model to rescue good steps. It reports 72.17% on AIME24 vs 70.50% for vanilla BCPG.

Both assume a verifier that already says which traces are wrong. They then refine credit within a trace. Our setting is the opposite: the target domain has no verifier, so the open problem is deciding at the answer level which outcomes are wrong at all. We borrow their motivating evidence that negative-only signals can be productive. The file reports that fine-tuning only on incorrect responses raised AIME24 from 10% to over 41%, and ResRL improves over negative-sample reinforcement. We do not borrow their machinery.

Transfer papers set up the starting condition. CPL trains only on math and reports +8.6% on GPQA and +12.2% on HumanEval. Satori reports transfer to FOLIO, StrategyQA and CRUXEval. Reasoning Vectors shows math GRPO gains behave like a transferable weight direction, with +12.3% on BBH at 1.5B. All three evaluate the target zero-shot and never adapt on it. We ask what happens after that zero-shot starting point.

Research Program

The starting model is Qwen2.5-7B trained with GRPO (a group-normalized policy-gradient method) on MATH and GSM8K, following the recipe the Reasoning Vectors paper uses. A second starting point is the released Satori-Qwen-7B checkpoint if it is available, to check that results do not depend on one training recipe.

The first phase is diagnostic and needs no training. On math, where labels exist, we measure how precise eliminations and majority votes are as a function of rejection rate and vote margin. We then repeat the measurement on each target, using gold labels for analysis only. This directly tests the asymmetry hypothesis that "no" judgments are more accurate than "yes" answers. It also tests whether the math-measured precisions predict the target-measured ones.

The second phase is adaptation. We run GRPO on unlabeled target training questions with the reward above. The elimination labels are recomputed at the start of each round from a frozen snapshot of the model, in the round-based style UPS uses, to slow feedback loops.

The target domains are chosen so that answer-set size varies. FOLIO has $K=3$, GPQA and ARC-C have $K=4$, and BBH mixes task-dependent $K$. CRUXEval-O has free-form outputs, effectively large $K$, and can be executed to get ground truth. Execution is used only to score results, never as a training signal. StrategyQA ($K=2$) is a built-in control. With two answers, eliminating one is the same as asserting the other, so the method should give no gain over positive self-training there. A gain on StrategyQA would mean the improvement comes from something other than the complementary mechanism.

Baselines

The naive reference point is the math-trained starting model evaluated zero-shot on each target, with self-consistency voting at inference to match the sampling budget. Every other method must beat it. The strongest existing transfer systems, CPL and Satori, belong to this zero-shot category. For FOLIO, the file reports GPT-4 8-shot at 64.2%, Flan-T5-Large fine-tuned at 65.9%, and Logic-LM (GPT-4 plus a symbolic solver) at 78.1%, all from FOLIO. There is no reported result found on Lacuna for Satori's individual FOLIO score, CRUXEval score, or StrategyQA score, and no absolute GPQA figure for CPL. We therefore rerun the zero-shot starting models ourselves under identical decoding.

The strongest competitor on this specific question is positive majority-vote self-training. It uses the same GRPO loop and target questions, rewards rollouts that match the confident majority answer, and has its margin threshold swept. There is no reported result found on Lacuna for majority-vote self-training on any of these reasoning targets. We run it with total inference compute equal to our method: we count elimination queries and give the baseline correspondingly more rollouts per question. Its margin threshold gets the same number of trials as our $\tau$.

A second competitor isolates the asymmetry claim. It uses the same elimination prompt but rewards rollouts whose answer the model judges plausible, turning self-verification into a positive reward. If this matches our method, the benefit comes from having a second view at all, not from trusting only rejections.

Three ablations isolate the remaining ingredients. The first is negatives-only, $w_+=0$. The second uses frequency-based exclusion in the NLNL style, penalizing answers that are rare among rollouts, with no elimination view. We expect it to hurt, especially on small $K$. The third is balancing by grid search over $w_\pm$ against a small labeled target development set. This last one is an unfair-but-informative ceiling: if our label-free weights come within noise of it, the balancing contribution holds.

Beating the strongest competitor means higher target accuracy than compute-matched majority-vote self-training, averaged over three seeds and outside seed variance, on at least three of the four $K \ge 3$ targets. It also means no more than one point of loss on MATH500 and GSM8K.

Why not just…?

Why not just self-train on majority votes with a high confidence threshold? This is the real threat. On easy targets, where the confident majority is usually right, it may perform as well as our method. The UPS ablation shows that confidence alone is a poor proxy for correctness in classifiers: error was 16.50% with confidence alone versus 9.46% with uncertainty filtering. A high threshold also discards exactly the hard, low-agreement questions. The experiments settle this by splitting results by initial vote agreement. Our method should win mainly on low-agreement questions, where positive labels are both scarce and noisy. We will also track answer entropy over rounds to detect collapse onto wrong majorities. If the advantage is flat across agreement levels, or absent, the asymmetry hypothesis is wrong, and the phase-one precision measurements will already have shown it.

Why not just use the transfer that already happens zero-shot? CPL and Satori show real out-of-domain gains without any target training, and zero-shot use may be the right choice when target questions are scarce. Their numbers, however, measure a fixed model that never uses the unlabeled target data available. The zero-shot baseline shows directly whether adaptation adds anything. A null result there would be informative in its own right, because it would say math-induced skill is as good as unlabeled self-training can make it.

Why not just use ResRL or BCPG-NSA step-level mining on the target? Both need to know which traces failed. BCPG-NSA additionally needs a proprietary judge and a process reward model, which would bring in external bias rather than resolve pseudo-label noise. They could be layered on top of our answer-level labels once we produce them, but they cannot replace them.

Why not just grid-search the positive/negative weights? Grid search needs labeled target data for selection, which breaks the unlabeled assumption. BCPG-NSA's sharp sensitivity suggests the optimum is narrow. We run the grid search anyway as the oracle ceiling, so the gap between source-calibrated weights and target-tuned weights is measured rather than argued.

Decisive Checks

Five results would confirm or refute the proposal. First, on each target, eliminations are more precise than majority votes at matched coverage, and precisions measured on math predict target precisions within a few points. If this fails, the weighting scheme has no basis. Second, the method beats compute-matched majority-vote self-training on the $K \ge 3$ targets, with the gain growing as initial agreement falls. Third, StrategyQA shows no gain, confirming the $K=2$ equivalence. Fourth, frequency-based exclusion does worse than elimination-based exclusion. Fifth, on CRUXEval-O, where execution gives exact precision measurements at large $K$, the complementary signal is at least as useful as on the multiple-choice targets, consistent with NLNL's $1/(K-1)$ argument.

If the asymmetry between rejecting and asserting does not appear in phase one, we stop there and report it. That would itself be an informative finding about how math-trained models judge their own answers.