Remembering Which Pseudo-Labels to Trust: Denoised Self-Training for Language Models in Unlabeled Expert Domains

General-purpose language models still fail in specialist fields. On the AppealCase benchmark, every one of 20 mainstream models scored below 50% F1 at predicting whether an appeals court would reverse a civil ruling. Open models score under 15% on university-level theorem problems. Expert labels are the scarce resource in these fields. Unlabeled expert text, by contrast, is plentiful: court filings, exam questions, clinical documents. The obvious remedy is self-training, where the model labels this text itself and retrains on its own answers. That recipe is known to collapse when the model starts out mostly wrong, because it learns its own mistakes and grows more confident in them. Computer vision ran into exactly this problem when adapting 3D object detectors from one lidar dataset to another. It solved the problem with a specific set of tools. A learned quality score replaces raw confidence. A per-example memory tracks pseudo-labels across rounds of training. Each pseudo-label is sorted three ways: trusted, known-bad, or ignored.

This note proposes moving that set of tools from 3D boxes to the reasoning traces of language models, where none of it has been tried. A trace's quality is judged in two ways. The first is a correctness predictor trained on source tasks that have automatic answer checkers. The second is how stably the trace's answer survives across self-training rounds. Ambiguous examples are kept out of training entirely rather than forced into a label. The claim is concrete. When starting accuracy on the target domain is low, this denoising should be the difference between self-training that improves and self-training that drifts. Simpler options are ruled out as explanations by ablation: majority voting within a single round, and confidence filtering.

Core Claim

ST3D adapts a 3D detector trained on Waymo to KITTI without KITTI labels. Its detector carries a small head that predicts box overlap quality, a score that is better than classification confidence at telling good boxes from bad ones. A memory bank keeps, for each target scene, the best pseudo-labels seen so far, and boxes are split into positive, negative and ignored zones. Together these lift detection average precision from 27.48 for the source-only model to 61.83, and to 73.37 with statistical normalization, close to the fully supervised 73.45. ST3D++ adds two pieces. One is voting over the memory's history when old and new pseudo-labels disagree. The other is source-assisted training with separate normalization statistics for each domain.

Our claim is that the structure of this solution transfers to language-model reasoning even though the data type does not. The analogue of the box-quality head is a probe on the model's hidden states. It predicts whether a reasoning trace reaches a correct answer, and it is trained on source tasks with programmatic verifiers. The analogue of the memory bank is a per-question record of pseudo-answers, quality scores and reasoning traces across rounds. The analogue of the three zones is a split into traces used for fine-tuning, traces used as rejected examples in a preference loss, and questions withheld from training this round.

The specific prediction is about where the method helps. Its advantage over naive self-training should be largest exactly where starting target accuracy is lowest. That regime is the "confirmation bias under severe shift" failure the vision literature identified, and it is the regime AppealCase and TheoremQA put open models in.

What Prior Work Motivates and Constrains

Two benchmark papers motivate the setting by showing a large gap that neither scale nor prompting closes. AppealCase builds 10,000 matched first- and second-instance Chinese civil judgments, balanced between affirmed and reversed. It reports that even reasoning-heavy models gain little. TheoremQA collects 800 expert-written STEM questions with objective numeric answers. GPT-4 with program-of-thought prompting reaches 51.0%, against 60–95% for undergraduates. Adding theorem definitions to the prompt gave no significant boost.

That second finding constrains the design. Supplying the missing knowledge as text is not enough; the model has to change how it reasons. This is why we update weights rather than prompts. It also means we do not borrow the text-descriptor prompting of DA-Pro. DA-Pro adapts detection heads with CLIP prompts that name the domain ("foggy"). That works for perceptual shift but conflicts with the TheoremQA result for reasoning.

Existing language-side adaptation either needs clean labels or uses weak filters. Pre-Training BERT on Domain Resources for Short Answer Grading inserts a stage of training on textbooks and curated question–answer pairs, and observes some loss on unrelated domains. It never uses unlabeled target data. Improving Low-resource Question Answering by Augmenting Question Information generates synthetic questions and found that without a correlation filter the noise hurt. This is direct evidence that pseudo-data quality control decides success in text too.

On the reinforcement-learning side, Reasoning Gym supplies more than 100 procedurally generated environments with automatic answer checkers. It shows that training on them transfers somewhat, with +9.7% on MATH from its math environments. We use it as the labeled, verifiable source domain. It constrains us too: such checkers do not exist for legal reversal, so the target must be handled without rewards.

$n$-Musketeers combines frozen expert models through reinforcement learning on verifiable rewards. It reports a warning we take seriously: adding machinery to an already-competent model dropped Reasoning Gym Logic from 96.88% to 82.81%. Nirvana adapts fast weights on each test sample with an online self-supervised objective. It is the closest attempt at on-the-fly specialization. Unlike Nirvana, our method does not adapt per sample. It accumulates evidence about each unlabeled example across rounds, and it explicitly refuses to learn from examples it cannot yet judge.

Research Program

The base model is an open instruction-tuned model with Chinese competence, such as the Qwen-2.5-3B-Instruct used by $n$-Musketeers, with a 7B variant as a scale check. Training uses low-rank adapters (LoRA).

The source domain is a fixed mixture of Reasoning Gym environments, on which every trace can be checked automatically. On source traces we train the quality probe: a small classifier on mid-layer and final-layer hidden states at the end of the trace, predicting verifier correctness.

Three unlabeled target domains are used, with their labels hidden during training and revealed only for evaluation and diagnostics. The first is AppealCase reversal prediction, the hardest regime. The second is TheoremQA. Its 800 items are split into a pool used as unlabeled training data and a held-out test half. The third is the pharmacist-exam questions of NLPEC from Towards Medical Machine Reading Comprehension with Structural Knowledge and Plain Text, where starting accuracy is moderate.

Each self-training round samples $k$ traces per unlabeled target question and scores each trace with the probe. It then updates the memory. A new pseudo-answer replaces the stored one only if its quality is higher. When the current and stored answers disagree, the final answer is decided by quality-weighted voting over the stored history. For trace $j$ of question $i$ in round $r$, the trust score combines the probe and the memory:

$$s_{ij}^{(r)} = q_{ij}^{(r)} \cdot \mathrm{stab}_i^{(r)},$$

where $q_{ij}^{(r)}$ is the probe's predicted probability that the trace is correct. The term $\mathrm{stab}_i^{(r)}$ is the fraction of the question's past rounds whose memory answer matches the trace's answer.

Traces with $s$ above an upper threshold become positives for supervised fine-tuning. Traces with high model probability but low $s$ become negatives, paired against a positive for the same question when one exists, in a preference loss. Everything between the thresholds is ignored.

Thresholds are set by class to keep the pseudo-label class ratio near the source prior. This matters on AppealCase, where a weak model may collapse toward predicting "affirmed." In the spirit of ST3D's curriculum augmentation, the unlabeled pool is admitted in order of document length, with harder items entering in later rounds. Following the source-assisted training of ST3D++, every update batch mixes verified source traces, and source and target get separate adapter normalization parameters. This keeps the model anchored and addresses the specialization–generalization trade-off.

Throughout, the hidden labels let us measure what self-training papers usually cannot: the precision of each zone in each round. This turns the mechanism into something we can inspect, not only an end score.

Baselines

The do-nothing reference is the base model prompted zero-shot with chain-of-thought on each target. Literature anchors frame the gap but are not run by us. On AppealCase, the paper reports that all 20 models fall below 50% F1. On TheoremQA, GPT-4 with program-of-thought reaches 51.0% accuracy and open models fall below 15%. On NLPEC, KMQA reaches 61.8% accuracy against a 60% passing line, using a supervised RoBERTa reader plus a medical knowledge graph KMQA. KMQA uses target labels, so it is an upper-reference point, not a fair unsupervised competitor.

The source-only baseline is the base model after LoRA training on the Reasoning Gym mixture with verifiable rewards, with no target data. This isolates how much of any gain is just generic reasoning transfer, of the kind Reasoning Gym already reports.

Naive self-training labels each target question with the model's single answer. It keeps answers above a confidence threshold and fine-tunes, repeated for the same number of rounds. This is the direct analogue of the baseline ST3D beat.

The strongest competitor is self-consistency self-training. It samples $k$ traces, takes the majority answer, keeps questions whose agreement exceeds a threshold, and fine-tunes on traces that reach that answer. It is cheap and widely used. In the binary and multiple-choice formats here it may capture most of what any denoising offers. No reported result found on Lacuna for self-consistency self-training on AppealCase, TheoremQA or NLPEC, so we establish its numbers ourselves.

Fairness rules apply to every self-training method. All get the same base model, the same unlabeled pool, the same $k$, the same number of rounds and the same total generated tokens. All also get the same source mixing. Thresholds are tuned on an identical grid using a 100-item labeled target development split shared by all methods; that split is excluded from test. Results are averaged over three seeds.

Ablations remove one ingredient at a time: the cross-round memory (stability term set to one), the probe (quality replaced by majority agreement), the negative zone (negatives discarded), and the ignored zone (every question receives a label). Beating the self-consistency baseline means three things. Test macro-F1 must be higher on AppealCase and test accuracy higher on TheoremQA by more than the seed spread. Positive-zone precision must be higher in matched rounds. And the memory-free ablation must lose most of the gain.

Why not just…?

Why not just use a bigger frontier model? The cited evidence says scale alone does not close these gaps. GPT-4 is among the AppealCase models below 50% F1. ChatGPT gets 20.8 exact match on PolicyQA, against 58.9 for a small BERT with targeted augmentation in PQQ. Our method is also orthogonal to scale, and the 7B run checks whether the gain shrinks as the base model gets stronger.

Why not just retrieve the statute or theorem into the prompt? TheoremQA reports that providing theorem definitions gave no significant boost. KMQA's error analysis attributes 32% of errors to retrieval noise. We include a retrieval-augmented prompting run on TheoremQA as a reference. The published result suggests it will not explain away weight-level adaptation.

Why not just do self-consistency self-training? This is the real threat, and we do not argue it away. Majority voting is a within-round version of our memory. If the model's errors are random, voting already removes them. Our bet is that under severe shift the errors are systematic, such as AppealCase models defaulting toward one outcome. Then a majority is confidently wrong, and only signals from outside the round can expose it: cross-round instability and a correctness probe trained where truth is checkable. The zone-precision diagnostics test this directly. If, at matched rounds, self-consistency positives are as precise as ours and end accuracy is within seed noise, the contribution reduces to "self-consistency suffices," and we would report it as such. If the advantage appears only on AppealCase and not on NLPEC, that would support the low-starting-accuracy hypothesis specifically.

Why not just train on source tasks with verifiable rewards and stop? Reasoning Gym shows real but partial transfer. That covers mathematical targets, not legal ones. The source-only baseline measures this directly. Any gain above it must come from the unlabeled target data.

Decisive Checks

Four results would decide the proposal.

The first is the precision trajectory. Plotting pseudo-label precision by round on AppealCase, naive self-training should stay flat or decline, and the full method should rise. This mirrors the confirmation-bias story from ST3D. The second is the main comparison against self-consistency self-training under matched compute.

The third is a correlation across the three targets. If the hypothesis is right, the method's margin over self-consistency should be largest where starting accuracy is lowest: AppealCase, then TheoremQA, then NLPEC. The fourth is a harm test prompted by the $n$-Musketeers negative-transfer result. We run the full pipeline with a target where the base model is already strong, the Reasoning Gym Logic environments with answers hidden, and confirm that accuracy does not fall. We also check that source and general reasoning accuracy stay within one point of the source-only model. We additionally report the probe's calibration on each target against hidden labels. If the probe transfers poorly, the result shows whether the memory term alone carries the method.