Where the Gradient Points: Low-Rank Subspace Agreement as a Pseudo-Label Filter

Self-training teaches a model with its own guesses on unlabeled data. It has become a cheap way to adapt large pretrained models, especially when paired with parameter-efficient fine-tuning, which freezes the pretrained network and trains only a small add-on such as LoRA (a low-rank correction $\Delta W = BA$ added to each weight matrix). Two literatures meet here and do not talk to each other. One works to close the "performance gap" between LoRA and full fine-tuning by making low-rank updates follow the full gradient more faithfully. The other works to stop models from learning their own mistakes. Our claim is that these goals pull against each other under noisy pseudo-labels. Making LoRA behave like full fine-tuning should also make it memorize wrong pseudo-labels faster. The same machinery that measures how far a low-rank update is from the full gradient can instead tell us which pseudo-labels to trust.

The concrete idea is simple. Estimate a reference low-rank subspace from the gradients of the small labeled set. For each pseudo-labeled example, measure what fraction of its gradient energy falls inside that subspace. Correct pseudo-labels should push the weights in directions the labeled data already agrees on. Wrong pseudo-labels should push in scattered, off-subspace directions. We use this "subspace agreement" score to decide which examples get full-gradient-faithful updates and which get down-weighted. The score needs no extra forward passes, no second model, no external relabeling model, and no meta-gradients.

Core Claim

The claim has two parts, each testable on its own. The first is diagnostic. Gradient-aligned LoRA variants close the performance gap on clean data, but in a self-training loop they shorten the window before the model begins fitting its own errors. The evidence behind this is indirect. LoRA-Pro matches LoRA's equivalent full-matrix gradient to the true gradient at every step and reports a GLUE average of 88.44 against 87.91 for full fine-tuning. LoFT projects the optimizer's moment estimates into the moving low-rank subspace and beats full fine-tuning on GPT-2 Large, which its authors attribute to implicit low-rank regularization. Neither paper tested training on labels the model generated itself. Self-training work, meanwhile, shows that early errors compound and permanently damage representations. Tri-net and Learning Transferrable Representations for Unsupervised Domain Adaptation both report this. If faster convergence carries equal speed on clean and noisy targets, then closing the gap is a double-edged improvement in this setting.

The second part is constructive. Let $G(x) \in \mathbb{R}^{d_{out} \times d_{in}}$ be the gradient of one frozen weight matrix with respect to the loss on example $x$ and its pseudo-label. Let $U$ and $V$ hold the top-$r$ left and right singular vectors of the labeled-set gradient, as in LoRA-GA but computed only on human-labeled data. The agreement score is

$$s(x) = \frac{\| U^\top G(x)\, V \|_F^2}{\| G(x) \|_F^2},$$

the fraction of the example's gradient energy that lies in the subspace the clean data defines. The score is cheap because a transformer layer's per-example gradient is a sum of per-token outer products $\delta_t x_t^\top$. Here $\delta_t$ is the backpropagated error at token $t$ and $x_t$ is the layer input. The numerator therefore reduces to an $r \times r$ computation, $\sum_t (U^\top \delta_t)(V^\top x_t)^\top$. The denominator needs one token-pair Gram product, $\sum_{t,t'} (\delta_t \cdot \delta_{t'})(x_t \cdot x_{t'})$. Both are affordable for ViT sequence lengths and short NLU inputs. The reference subspace is refreshed by an exponential moving average over the labeled gradients plus the currently high-scoring pseudo-labeled gradients. Only examples above a score threshold, or weighted by $s(x)$, receive the aligned LoRA-Pro or LoFT update. Everything else is down-weighted.

Relation to Prior Work

Four papers already join parameter-efficient adaptation and pseudo-label noise. None of them uses gradient geometry as the reliability signal. TriNoL routes pseudo-labeled images to three LoRA experts by softmax confidence and trains them with plain AdamW. It motivates our work by showing that mixed clean and noisy gradients hurt a single adapter. It also exposes the weakness we target: its routing trusts confidence, which it admits fails under miscalibration and domain shift. UPET filters pseudo-labels with Monte Carlo dropout, which requires many stochastic forward passes, and uses hard examples as contrastive negatives. It supplies our main NLU testbed and a strong uncertainty baseline, and our score replaces its repeated forward passes with one backward pass. LiST reweights pseudo-labels by meta-gradients on the labeled set. That is the closest in spirit, since it also asks whether a sample's gradient helps the labeled loss, but it pays for second-order computation. We ask a first-order, subspace-level version of the same question. Delora trains a clean and a "noisy" LoRA to detect mislabeled text offline and then relabels with GPT-4o. It shows that parameter-level separation detects noise far better than small-loss selection, with 99.47% detection precision on Trec against roughly 80%. It is an offline cleaner rather than an online self-training loop, and it depends on a frontier model.

The gap-closing methods play the role of supplied technique, not competitor. We reuse LoRA-GA's gradient SVD, LoRA-Pro's equivalent-gradient view, and LoFT's optimizer projection. The difference is that we apply them only to the examples that the subspace test admits. Uni-UVPT handles noise by timing a label-correction moment in dense segmentation. We treat that temporal strategy as a baseline family rather than extending it, and dense prediction is out of scope for this note.

Research Program

The work has three stages. The first stage is a controlled diagnosis. Using semi-supervised benchmarks where the hidden labels of "unlabeled" data are known, we track three quantities over training for vanilla LoRA, LoRA-GA, LoRA-Pro and LoFT at matched rank. These are the fraction of wrong pseudo-labels the model has fit, the pseudo-label error rate of each self-training round, and final accuracy. The prediction is that aligned methods fit wrong pseudo-labels earlier.

The second stage validates the agreement score purely as a detector. We measure its AUROC for separating correct from incorrect pseudo-labels and compare it with softmax confidence, loss, raw gradient norm, and MC-dropout variance. We also break results out by noise type. The expected failure case matters here. Systematic confusions, such as two similar bird species, may produce wrong-label gradients that are themselves low-rank and consistent. The score may miss exactly these. We will report this honestly rather than average it away.

The third stage runs the full loop. Admitted examples get aligned updates and the rest are down-weighted. An optional variant sends low-score examples to a negative-learning loss in TriNoL's style, which tests whether our score is a better router than confidence.

Proposed Experiments

In vision, we follow TriNoL's setting with CLIP and DINOv2 ViT backbones on FOOD-101, Semi-Aves, and CIFAR-100. Semi-Aves matters most because its unlabeled pool mixes in out-of-distribution images, where confidence is least reliable. In language, we follow UPET's 16-shot protocol with RoBERTa-large on its seven NLU tasks. We also run a Llama-class 7–8B model with LoRA on a subset to test whether the effect survives in the regime where gradient-aligned methods were developed. As a pure detection check, we use Delora's synthetic-noise text classification (Trec, SST-2, AGNews at 20% and 40% symmetric and asymmetric noise). This isolates the score from the self-training loop. The metrics are top-1 or task accuracy, pseudo-label precision per round, detection AUROC, memorization curves, and wall-clock time per iteration.

Baselines

The naive reference point is single-adapter vanilla LoRA with confidence-thresholded self-training. It uses the same backbone, rank, pseudo-labeling schedule and augmentation as every other arm, and it shows what the extra machinery buys at all. The second reference point is the same loop run with LoRA-Pro or LoFT in place of vanilla LoRA and no noise handling. This is what a practitioner would try after reading the gap-closing literature. Beating it tests the central tension: our method must match its speed on clean data while ending with higher accuracy and lower pseudo-label error.

The strongest vision competitor is TriNoL. It is the only method built for this exact regime, and it reports 88.42% top-1 on FOOD-101 against 87.27% for FineSSL. On Semi-Aves it reports a margin of "over 0.5%" without an absolute number. We found no reported result on Lacuna for TriNoL on CIFAR-100 or ImageNet, so we will rerun it with its three rank-8 experts at matched total parameter count and our tuning budget. Beating it means exceeding its accuracy on FOOD-101 and Semi-Aves across three seeds with non-overlapping intervals, at comparable per-iteration cost; TriNoL reports 0.218s per iteration against 0.182s for single LoRA. The Semi-Aves out-of-distribution split is where we expect any gain to show.

The strongest language competitor is UPET, which reports 78.2% average 16-shot accuracy against 76.2% for LiST and 74.6% for standard self-training. We rerun UPET with its own number of dropout passes and the same parameter-efficient module. Beating it means a higher average at equal or lower teacher compute. For pure detection, Delora's 99.47% precision on Trec is the bar. We compare only its first, detection stage, without GPT-4o relabeling, so that both methods draw on the same internal signal.

The ablations isolate the key ingredient. We replace $s(x)$ with raw gradient norm, with loss, and with confidence, keeping everything else fixed. We also compute the reference subspace from pseudo-labeled data instead of labeled data. Finally, we pair the score with a vanilla rather than an aligned optimizer, which separates the benefit of better filtering from the benefit of better optimization. All arms get the same hyperparameter search size and the same seeds.

Why not just…?

Why not just threshold on confidence and keep vanilla LoRA? UPET reports that removing its uncertainty-based selection costs over 2% across its NLU tasks. TriNoL names confidence miscalibration as its own routing weakness. Still, confidence thresholding is cheap. If it matches our score's AUROC on Semi-Aves's out-of-distribution split, our method has no reason to exist. The second-stage detection comparison settles this directly.

Why not just use per-sample loss or gradient norm, which need no reference subspace? This is the most serious threat. A large gradient norm already flags examples the model disagrees with, and it may carry most of the signal. Delora's finding that small-loss selection only reaches about 80% precision argues against loss. There is no reported result on Lacuna for gradient-norm filtering in this setting, so it enters as a direct ablation. If the norm-only score comes within noise of $s(x)$, we will report that the subspace adds nothing and recast the contribution as the diagnostic finding alone.

Why not just rely on low rank as the regularizer and skip alignment entirely? LoFT's GPT-2 Large result and LoRA-Pro's MRPC result both hint that low rank resists noise. If vanilla LoRA at small rank already beats aligned variants in self-training, that confirms our diagnostic claim. It would also suggest the right move is filtering plus vanilla updates, which the optimizer ablation measures. Raising rank instead looks unpromising: TriNoL's matched-parameter single rank-24 LoRA lost to its routed experts.

Why not just route by confidence as TriNoL does, since multiple experts already separate the noise? Routing and scoring are complementary. The variant that feeds $s(x)$ into TriNoL's three experts tests whether the router, not the architecture, is the bottleneck.

Decisive Checks

The proposal stands or falls on three results. The first is whether aligned LoRA variants measurably speed up memorization of wrong pseudo-labels compared with vanilla LoRA. The second is whether subspace agreement beats confidence, loss, and gradient norm at detecting wrong pseudo-labels, especially on out-of-distribution unlabeled data. The third is whether gating aligned updates by that score beats TriNoL on Semi-Aves and UPET on 16-shot NLU at no greater compute. A positive first result with a negative second one is still publishable: it would be a cautionary finding for the gap-closing literature. A negative first result means the tension we posit does not exist, and the method must justify itself purely as a cheaper detector.