Clip-Gated Teachers: Letting Curvature Decide When a Self-Training Teacher Listens

Many label-efficient systems train on a small labeled set plus a large unlabeled pool. A slowly moving "teacher" copy of the network labels the unlabeled data, and a fast "student" copy learns from those guesses. The teacher is an exponential moving average (EMA) of the student's weights: at every step it absorbs a fixed small fraction of the student. Fixing that fraction treats every step as equally trustworthy, and they are not. Some steps are quiet. In others the student is hit by a burst of confidently wrong pseudo-labels, its gradient norm jumps, and the optimizer has to clip. Optimization theory has a precise account of that second kind of step. Under generalized smoothness, the local curvature of the loss grows with the gradient norm, so the region where a step can be taken safely shrinks exactly when gradients spike. Our proposal is to let the teacher absorb less of the student in proportion to how hard the student's optimizer is clipping. The clip ratio is a quantity the optimizer already computes, so the gate costs nothing extra.

The claim is narrow and can be tested. A teacher whose per-step averaging weight is scaled by the student's clip ratio $r_t = \min(1, c/\|g_t\|)$ resists confirmation bias, meaning the loop where the model reinforces its own wrong labels. It should do so better than a fixed-decay teacher tuned to its best value, and at least as well as the existing uncertainty-gated and lag-corrected teachers. It should also be far less sensitive to the decay hyperparameter across label ratios. Two conditions must hold for this to work. Gradient-norm spikes during self-training must precede pseudo-label corruption, and the student must be allowed to keep its high learning rate while the teacher is insulated. We test both conditions directly before building anything on them.

What the three directions contribute, and what is missing

Generalized smoothness supplies the mechanism. Why Gradient Clipping Accelerates Training introduced the $(L_0, L_1)$ condition, under which the Hessian norm is at most $L_0 + L_1\|\nabla f(x)\|$. It showed that the safe step scales like $1/(L_0 + L_1\|\nabla f\|)$, and that clipping implements exactly this. Convex analyses such as Convergence of Clipped SGD on Convex (L0,L1)-Smooth Functions and Methods for Convex (L0, L1)-Smooth Optimization give clipped SGD rates that do not depend on a global smoothness constant.

Revisiting Gradient Clipping constrains the design. It proves that stochastic clipping carries a bias floor of order $\sigma^2/c$, where $\sigma$ is the gradient noise level and $c$ the clip threshold. So we do not claim that gating the teacher removes clipping bias. We claim only that it changes which iterates the teacher inherits.

The closest theory paper is Adam with model exponential moving average is effective for nonconvex optimization. It shows that a uniformly discounted EMA arises naturally on top of a clipped Adam. That analysis uses fixed discounting, does not use the $(L_0, L_1)$ condition, and never considers a loop in which the averaged model generates the student's training targets.

Temporal weight averaging supplies the object we modify, and its literature motivates the gap. Kaizen found a "Goldilocks" teacher half-life of at least about 1,000 updates for semi-supervised speech recognition. That horizon is measured in steps and has to be retuned. Revisiting Class Imbalance for End-to-end Semi-Supervised Object Detection observed the opposite failure: a fixed EMA lags when weights move rapidly on rare classes. It answered with a double EMA that makes the teacher respond faster. AdaSemiCD pauses teacher updates when a pseudo-label uncertainty metric rises.

These three papers pull in different directions, toward slower, faster, and gated teachers. None of them ties the teacher's rate to the student's optimization geometry, and none has a theoretical account. Exponential Moving Average of Weights in Deep Learning supplies a key enabling fact: EMA lets the student keep a high learning rate without diverging. It also flags two practical traps. Batch-normalization statistics desynchronize from averaged weights, and Kaizen adds that half-precision accumulation freezes the teacher.

Label-efficient learning is the application regime. Domain pretraining such as Advancing Radiograph Representation Learning with Masked Record Modeling and Multi-Granularity Cross-modal Alignment already gets far on 1% labels. Their evaluations stop at fine-tuning. The file notes that the next step, self-training on the unlabeled downstream pool, is where fine-tuning instability and confirmation bias appear. That step is where a geometry-aware teacher should matter most.

Core Claim

Let $\theta^s_t$ be the student weights, $g_t$ the student's minibatch gradient, and $c$ the clip threshold. The gated teacher update is

$$\theta^{\text{teach}}_{t+1} = \theta^{\text{teach}}_t + (1-\alpha)\, r_t^{\,p}\, \big(\theta^s_{t+1} - \theta^{\text{teach}}_t\big), \qquad r_t = \min\!\left(1, \frac{c}{\|g_t\|}\right).$$

Here $\alpha$ is the base decay and $p \ge 0$ is a sharpness exponent. Setting $p = 0$ recovers standard EMA.

Under the $(L_0, L_1)$ condition, $r_t$ is a cheap proxy for the ratio between the safe step and the nominal step. This gives the rule a principled reading. The teacher averages over a fixed budget of low-curvature student time rather than a fixed number of steps.

The theoretical target is modest and stated as a conjecture to be proved or refuted. In the deterministic convex $(L_0, L_1)$ setting, clipped steps occur only where $\|\nabla f\| > c$. These are exactly the steps that the existing analyses bound most weakly. Down-weighting them in a weighted iterate average should give a suboptimality bound with no dependence on the initial gradient norm. It should also have a better constant than uniform averaging, because the bound becomes a weighted sum dominated by unclipped steps. The stochastic version must account for the $\sigma^2/c$ floor. We would state the result relative to that floor, not as an escape from it.

Research Program

Diagnostic. The first stage asks whether the premise holds on modern self-training trajectories. We run a standard mean-teacher pipeline on three setups:

  • a ViT backbone pretrained with masked record modeling, self-training on CheXpert with 1% labels;
  • the AdaSemiCD change-detection setup on LEVIR-CD with 5% labels;
  • continual test-time adaptation on CIFAR10-C, following CoTTA.

Along each trajectory we log the gradient norm and an estimate of local smoothness, taken as finite differences of gradients along the update direction. We check whether the affine relation between curvature and gradient norm, which Zhang et al. reported for LSTMs, also appears here.

We then measure lagged correlations between gradient-norm spikes and three downstream events. The first is pseudo-label flips on a held-out unlabeled probe set. The second is teacher accuracy drops on a small labeled validation split. The third is the AdaSemiCD uncertainty metric. If spikes do not lead corruption, or if they carry no information beyond uncertainty, the proposal is weakened or reduced to AdaEMA, and we would report that.

Method study. The second stage implements the gated teacher in all three pipelines. We sweep $p$ and $\alpha$, keep EMA accumulation in fp32, and use LayerNorm backbones or recompute batch-norm statistics so that the batch-norm failure mode cannot confound the comparison.

Theory. The third stage attempts the weighted-averaging bound. It also checks that bound against synthetic $(L_0, L_1)$ problems with injected label noise. On those problems, ground-truth curvature is known.

Baselines

The naive reference point is the student fine-tuned on the labeled fraction alone. It establishes how much self-training helps at all. AdaSemiCD reports this baseline at 61.0% IoU of changed pixels on LEVIR-CD with 5% labels. The standard pseudo-labeling reference on CIFAR100-C has a 46.2% error, from Continual Test-Time Domain Adaptation.

The main control is a fixed-decay EMA teacher. Its decay is tuned by a Kaizen-style sweep over half-lives, which is the honest version of "just tune $\alpha$." We give it the same tuning budget as the gated teacher, counted in total configurations. We report two numbers for both methods: the best result, and the spread across label ratios when the decay is held at the value tuned for a single ratio.

The strongest existing competitors are domain-specific. On LEVIR-CD the strongest is AdaSemiCD's uncertainty-gated AdaEMA, reported at 77.7% IoU on changed pixels with 5% labels. We run it from its described configuration and swap only the teacher rule. On CIFAR10-C and CIFAR100-C the strongest is CoTTA, reported at 16.2% and 32.5% average error. We compare both against full CoTTA and against CoTTA with its EMA replaced by the gated rule. The second comparison tests whether gating can stand in for part of the cost of its 32-augmentation averaging. Double EMA from the semi-supervised detection paper is included as the lag-correcting alternative. On CheXpert 1% there is no reported self-training result on Lacuna for a self-trained MRM or MGCA backbone. The reference is MRM's 88.5% mean AUC from fine-tuning with 1% labels. Self-training must beat that number to matter at all.

Three ablations isolate the key ingredient:

  • Random gate. Its average absorption matches the clip gate but its timing is random. This tests whether when the teacher listens matters, or only how much it listens overall.
  • Displacement-weighted EMA. The weights grow with the actual step length $\eta\min(\|g_t\|, c)$. This is the opposite hypothesis: the teacher should follow the student's motion.
  • Uncertainty-signal gate. The same functional form, driven by the uncertainty signal instead of the clip ratio.

Beating the competitors means one of two outcomes. The first is a gain over tuned fixed EMA and over AdaEMA or CoTTA beyond seed variance on at least two of the three benchmarks. The second is matching them while cutting the cross-ratio degradation of an untuned decay by half.

Why not just…?

Why not just tune the EMA decay carefully? Kaizen shows that a good fixed horizon exists. If one value transfers across label ratios, learning rates, and domains, our gate adds nothing. The fair test is the tuning-budget-matched comparison above, plus the transfer protocol in which the decay is tuned at one label ratio and evaluated at the others. The double-EMA paper's observation of lag on rare classes suggests a fixed horizon is wrong at specific moments, not on average. But this is the most likely way the proposal fails, and the transfer results will decide it.

Why not just clip the student harder? Harder clipping shrinks every step that has a large gradient. That makes the student move less in exactly the phases where it may be learning rare classes, and it slows learning overall. Morales-Brotons et al. found that EMA's value comes partly from letting the student keep a high learning rate. Gating moves the conservatism from the student to the teacher. We test this by sweeping the student's clip threshold with an ungated teacher. If some threshold matches the gated teacher's accuracy at similar wall-clock time, this alternative wins and we would say so.

Why not just use AdaEMA's uncertainty gate? This is the real threat. If gradient spikes and uncertainty spikes coincide, the two gates are nearly the same method. Ours would then be only cheaper: it needs no per-pixel uncertainty maps and applies to any task, including speech and continual adaptation, where AdaSemiCD's metric does not exist. The diagnostic stage measures the overlap directly. The uncertainty-signal ablation compares the two signals within the same update rule, so any advantage attributable to the clip ratio, or its absence, will be visible.

Decisive Checks

The idea stands or falls on three results. The first is the diagnostic: gradient-norm spikes must lead pseudo-label flips with a lag that a teacher could exploit. The second is the random-gate ablation: it must underperform the clip gate, which would show that timing carries the benefit. The third is the transfer protocol: the gated teacher must keep its accuracy across label ratios where tuned fixed EMA does not.

A secondary check runs on CIFAR-100N, which has real human label noise. There, standalone EMA is reported at 65.15% against 55.50% for SGD. A gated EMA should widen that gap if wrong labels really show up as clipped steps. If the first check fails, we report the measured curvature–gradient relation on self-training trajectories as a standalone finding. We would then drop the method claim.