A Learned Feedback Controller for Pseudo-Labeling Hyperparameters

Semi-supervised learning trains a classifier from a few labeled examples plus many unlabeled ones. Most current methods do this by pseudo-labeling: the model labels its own unlabeled examples and trains on the guesses it is confident about. Two knobs decide how well this works. One is the confidence threshold that decides which guesses are kept. The other is the weight given to the unlabeled loss. Today these knobs are either fixed, set by a decay formula chosen before training, or adjusted by hand-written rules based on the model's own predictions. None of these choices can notice that training is going wrong, for example when the model starts confidently mislabeling a rare class or soaking up junk unlabeled images, and then reverse course. We propose to learn a small controller that does notice. Every few hundred steps it reads cheap statistics that need no labels, such as how many pseudo-labels each class is accepting, whether the predicted class mix is drifting away from the labeled class mix, and how often weakly and strongly perturbed views of an image disagree. From these it sets the threshold and the unlabeled-loss weight for the next stretch of training. The controller is trained once, offline, on many semi-supervised runs where we secretly have full labels and can score its decisions. It is then deployed on new datasets with no further tuning.

The motivation comes from reinforcement learning. There, work on automated hyperparameter tuning has shown that the best setting moves during a single run, and that framing scheduling as a sequential decision problem beats any single fixed value. Semi-supervised learning has the same kind of feedback loop: the model's own predictions change the data it trains on. Yet its dynamic schedules are still mostly open-loop formulas. The question this note asks is narrow and testable. Does closing the loop with a controller that is learned once and then transferred give gains that open-loop schedules, hand-written feedback rules, and brute-force population search do not? The test cases are class imbalance and out-of-distribution unlabeled data, which are exactly where existing dynamic thresholds are known to break.

Core Claim

Our claim has three parts. First, label-free training statistics carry enough information to detect confirmation bias, the runaway process in which wrong confident pseudo-labels reinforce themselves. Second, a policy mapping those statistics to threshold and loss-weight adjustments can be learned offline with privileged labels on source tasks. Third, that policy transfers across datasets and modalities, so on a new task it needs neither a labeled validation set nor a meta-tuning budget.

The novelty lies in where the feedback signal comes from and in the fact that the controller is trained once and reused. It is not simply the idea of making hyperparameters dynamic, which Dash and FlexMatch already do.

Relation to Prior Work

Three semi-supervised methods make thresholds dynamic, and they motivate this work by exposing its failure modes. Dash replaces FixMatch's fixed confidence cutoff with a loss threshold. That threshold decays geometrically from a value estimated during a labeled warm-up. It is open-loop: once the decay constants are chosen, it cannot respond to what training does. FlexMatch does respond, lowering thresholds for classes that have accepted few confident predictions. But its signal is the model's own counts, so on imbalanced data such as SVHN a corrupted count keeps thresholds low and noise accumulates. FixMatch itself needs a distribution-alignment fix on CIFAR-100. We read these three results as evidence that the right feedback signal is not a single self-count. It is a combination of statistics that can disagree with one another when the model is fooling itself.

Four meta-learning and RL-flavoured methods supply competing mechanisms. They also constrain our design, because they show that labeled-set feedback is expensive or myopic. Semi-Supervised Learning with Meta-Gradient adjusts soft pseudo-label vectors by differentiating the labeled loss through one inner step. Meta-Semi weights each unlabeled sample by how well its gradient aligns with the labeled gradient. RLGSSL treats pseudo-label assignment as a single-step bandit with a Mixup reward. SemiReward learns a network that scores individual pseudo-labels but still applies a fixed acceptance cutoff. All four use the tiny labeled set as their online signal, and all four decide one step or one sample at a time. Our controller acts on global knobs over a multi-step horizon. On the deployed task it never consults labels beyond the training set, which matters when there are only four labels per class to begin with.

The method itself comes from reinforcement learning hyperparameter tuning. Trust Region Meta Learning for Policy Optimization treats training as a meta-decision process. Its state is a set of statistics such as entropy and policy divergence, its action is a step-size parameter, and its reward is the resulting improvement. It fits the policy offline with Fitted-Q Iteration, a batch method that repeatedly regresses action values on logged transitions, and the learned schedule generalized across policy architectures. We reuse that recipe almost directly. AutoRL Hyperparameter Landscapes and STAC supply the evidence that optimal settings move and that learned trajectories are non-monotonic. Non-monotonic means raising a value and later lowering it again, which no decay formula can do. The AutoRL survey identifies the meta-hyperparameter problem: online tuners add knobs of their own. Training offline and transferring the result is our answer to that problem, because the meta-knobs are paid for once across source tasks rather than once per task.

Finally, USB makes this feasible and defines the setting. Fine-tuning pre-trained ViT, BERT and Wav2Vec 2.0 backbones cuts semi-supervised training cost by roughly 90%, which makes logging hundreds of training runs affordable. USB also reports that several semi-supervised methods fall below supervised training when labels are scarce, and that is the failure we want the controller to catch.

Research Program

The base learner is FixMatch-style training inside the USB codebase, which the SemiReward implementation also builds on. Training is divided into decision intervals of about 2% of the run.

At the start of each interval the controller observes a state vector $s_k$, where $k$ indexes the interval. The vector contains:

  • per-class acceptance rates;
  • the divergence between the class distribution of accepted pseudo-labels and the labeled class distribution;
  • the entropy of the averaged prediction;
  • weak-versus-strong view disagreement;
  • disagreement between the student and its slowly updated average copy, the teacher;
  • the mean unlabeled loss;
  • the fraction of training elapsed.

It then picks a discrete action $a_k$. The action multiplies the global threshold, scales the unlabeled-loss weight, and optionally applies a shared offset to thresholds for under-accepting classes.

On source tasks, the reward is the change in accuracy on a large held-out labeled pool over the interval:

$$r_k = \mathrm{Acc}_{\text{held-out}}(\theta_{k+1}) - \mathrm{Acc}_{\text{held-out}}(\theta_k),$$

where $\theta_k$ is the model at interval $k$. This held-out pool exists only on source tasks and is never available to the deployed controller.

The source tasks are USB tasks augmented with stress variants: label budgets from 1 to 25 per class, long-tailed subsampling of the unlabeled pool, and injected out-of-distribution unlabeled data from other USB datasets. Logged trajectories come from two kinds of behavior policies: randomized schedules, and perturbed versions of Dash and FlexMatch schedules. We fit an action-value function with Fitted-Q Iteration, then run a second collection round with the learned controller to reduce the mismatch between the logged data and the states the controller itself visits. The deployed controller is a lookup over a small model with negligible cost.

The main analysis is held-out transfer. We train the controller on all tasks except one dataset, or except one entire modality, and evaluate on what was held out.

Baselines

The naive reference point is FixMatch with its fixed threshold of 0.95 and fixed loss weight, run in USB with the same backbone, iteration budget and augmentations as every other method. On CIFAR-10 with 40 labels, FixMatch reports 88.61% accuracy in the from-scratch Wide-ResNet setting. A second naive point is supervised fine-tuning on the labeled set alone. USB shows this sometimes beats semi-supervised methods, and any method falling below it fails outright.

The strongest existing dynamic-threshold competitors are Dash and FlexMatch. In the from-scratch setting, Dash reports 9.16% error on CIFAR-10 with 40 labels, against 11.39% for its FixMatch reproduction, and 3.03% against 3.96% on SVHN with 40 labels. FlexMatch reports error reductions over FixMatch of 13.96% on CIFAR-100 and 18.96% on STL-10, both at 4 labels per class. SemiReward reports 1.9% to 3.7% accuracy gains on USB-style tasks with pre-trained backbones. For per-task Dash or FlexMatch numbers under USB's pre-trained protocol, we found no reported result on Lacuna, so we will rerun all of them ourselves.

Fairness rule for these competitors: each gets its published defaults plus a tuning budget equal to the number of runs the controller consumed on source tasks. That tuning is spent on the same source tasks and transferred, never tuned on the held-out task. Beating them means lower mean error on held-out tasks across at least five seeds and label draws. The gain must be largest on the imbalanced and out-of-distribution variants, and must never fall below supervised fine-tuning.

The compute-matched brute-force competitor is population-based training over the same two knobs. The population of the brute-force competitor copies and mutates hyperparameters during training (SEARL, Bayesian Generational PBT). Its selection criterion is a validation split carved from the labeled set. It is given a total compute budget on the held-out task matched to the controller's amortized cost, and also a version with eight times that budget.

The ablations isolate the key ingredients. A hand-written proportional rule reads the same state features. For example, it raises the threshold when the class-mix divergence grows and lowers the loss weight when view disagreement spikes. This tests whether learning the controller matters or only the features do. An open-loop replay runs the controller's average action sequence without feedback. This tests whether closing the loop matters at all. Removing groups of state features shows which signals carry the confirmation-bias information.

Why not just…?

Why not just use a good fixed schedule, such as Dash's decay or cosine annealing, tuned on source tasks? If the optimal schedule is roughly the same across tasks, a transferred open-loop schedule captures most of the benefit at no cost. The open-loop replay ablation tests exactly this. If replaying the controller's average actions matches the closed-loop controller on held-out tasks, feedback adds nothing and we will say so. We expect the gap to appear mainly on imbalanced and out-of-distribution variants, because there the right action depends on what the model does rather than on elapsed time.

Why not just tune hyperparameters online with population-based training on a labeled validation split? With 4 labels per class, a validation split is either tiny and noisy or eats into the training labels. Populations also multiply compute, a cost the AutoRL survey highlights. This competitor is real when compute is unlimited, which is why we include the eight-times-budget version. If it wins only at that budget, the controller's claim narrows to amortized efficiency.

Why not just write a feedback rule by hand, like FlexMatch plus distribution alignment? This is the most serious threat, because the learned controller may simply rediscover a simple rule. The hand-written proportional rule on the same features is built to settle this. If it matches the learned controller, the contribution becomes the state signals and the rule, which is still useful and cheaper. The learned policy would then be dropped from the claim.

Why not just use per-sample meta-reweighting such as Meta-Semi or SemiReward? These operate at a different level: they score individual samples, while our controller sets global knobs. They also depend on labeled gradients or a fixed cutoff. They are complementary, so we additionally run the controller on top of SemiReward, setting SemiReward's reward cutoff, to test whether the gains stack.

Proposed Experiments

All experiments use USB's pre-trained backbones on its CV, NLP and audio tasks. We use CIFAR-100, STL-10 with its distractor unlabeled images, and long-tailed and out-of-distribution variants of these, at 1, 4 and 25 labels per class. Error rate is the primary metric. We also report pseudo-label precision over time, and the fraction of runs falling below supervised fine-tuning, which measures robustness to the collapse USB reports.

The first decisive check is leave-one-modality-out transfer: does a controller trained on CV and NLP beat Dash and FlexMatch on audio?

The second is the imbalance test: on long-tailed unlabeled pools modeled on FlexMatch's SVHN failure, does the controller raise thresholds for the dominant pseudo-label class where FlexMatch lowers them? Does its pseudo-label precision then stay flat rather than decaying?

The third is the pair of feedback ablations: open-loop replay and the hand-written rule. Together they determine whether the claimed mechanism, learned closed-loop control, is what produces any gain.

A secondary from-scratch Wide-ResNet run on CIFAR-10 and SVHN at 40 labels checks consistency with the published Dash and FixMatch numbers above.