Protecting the Replicable Part of a Scarce-Label Gradient

When labels are scarce, a common trick is to train one network on two objectives at once. The first is the supervised task you care about, fit on a few hundred labeled images. The second is an auxiliary "self-supervised" task, such as predicting how an image was rotated, which needs no labels and can use the whole unlabeled pool. The auxiliary task shapes the shared features, but it can also pull them away from what the main task needs. A family of "gradient surgery" methods addresses this. When the two tasks' update directions disagree, they remove the disagreeing component. This proposal starts from an observation that breaks those methods in exactly the label-scarce regime: the main task's gradient, computed from a handful of labeled examples, is mostly noise and memorization. Deciding that the auxiliary task "conflicts" with it is close to a coin flip, and protecting it means protecting overfitting.

The idea is to protect only the part of the main-task gradient that replicates across disjoint subsets of the labeled data. We split the small labeled set into shards and compute each shard's gradient. This is cheap, because the labeled set is tiny. We keep the direction the shards agree on as a reference. The auxiliary gradient is then projected off that reference whenever it opposes it, and the main-task gradient itself is never modified. The bet is that in the label-scarce regime, what you project against matters more than whether you project. A replicated reference should turn gradient surgery from a no-op or a harm into a reliable gain, with no tuned loss weight.

Core Claim

The claim has three parts. First, at roughly 1–4% labels, the sign of the minibatch inner product between the main-task and auxiliary gradients agrees poorly with the sign computed against a low-noise estimate of the generalizing main-task gradient. Second, projecting the auxiliary gradient against a cross-shard replicated reference reduces held-out main-task loss more than projecting against the raw minibatch gradient. Third, the resulting single-stage method matches or beats tuned fixed-weight joint training at equal compute, using a weight-free rule.

Concretely, let the labeled set be split into $K$ disjoint shards. Let $\bar g_k$ be an exponential moving average of the main-task gradient on shard $k$, refreshed every few hundred steps. We define the reference as the shard mean, kept only when the shards agree:

$$g_{\text{ref}} = \frac{1}{K}\sum_{k=1}^K \bar g_k \quad \text{if} \quad \min_{k\neq l}\cos(\bar g_k,\bar g_l) > \tau, \ \text{else } g_{\text{ref}} = 0,$$

where $\tau$ is an agreement threshold. The auxiliary gradient $g_{\text{aux}}$ is then modified only when it opposes the reference:

$$g_{\text{aux}} \leftarrow g_{\text{aux}} - \frac{\min(0,\ g_{\text{aux}}\cdot g_{\text{ref}})}{\|g_{\text{ref}}\|^2}\, g_{\text{ref}}.$$

The shared parameters are updated with the untouched minibatch main-task gradient plus this filtered auxiliary gradient. A coordinate-wise variant keeps only reference coordinates whose sign is consistent across shards. It will be tested as an alternative to the cosine gate. When the shards disagree, which typically happens late in training once the labeled set is memorized, the reference vanishes and the auxiliary task trains freely. That is the intended behavior: an auxiliary task should be allowed to fight memorization.

Relation to Prior Work

Three papers motivate the gap. S4L trains supervised classification jointly with rotation or exemplar self-supervision on 1% and 10% ImageNet. It shows the two signals are complementary and that weights can be tuned on one validation image per class. However, its weights are fixed and it never inspects gradient interference. Boosting Few-Shot Visual Learning With Self-Supervision adds rotation or patch prediction to few-shot base training, also with a static weight. Both establish that auxiliary self-supervision helps under scarcity. Neither asks when it hurts.

Four papers supply or constrain the method. Gradient Surgery for Multi-Task Learning (PCGrad) projects conflicting gradients symmetrically, treating all tasks as equals. It was evaluated with full labels on every task. TGP-MTL targets a clinical primary task with an auxiliary speaker task. It projects the primary gradient off the auxiliary one, which bends the primary update to accommodate the auxiliary task. We do the opposite and never touch the primary update. AMGS uses a cosine-sign gate between support and query gradients inside meta-learning and discards conflicting updates entirely. Our gate is between tasks rather than episodes, and it projects rather than discards. GradNorm equalizes training rates across tasks. That rule is wrong-headed here: a slow auxiliary task would be up-weighted against a primary task we care about more.

All four compute conflict against the current minibatch gradient. None considers that with a few labels per class this gradient is dominated by sampling noise. That is the one ingredient we change.

Two related approaches are not reused. Deep-AMTFL protects scarce tasks by architectural reconstruction anchored on sample-rich tasks, which do not exist here. MAXL learns auxiliary labels with second-order meta-gradients, at a cost our first-order rule avoids.

Research Program

The first stage is diagnostic and can falsify the premise cheaply. On CIFAR-100 with 4, 25, and 100 labels per class, we train supervised-plus-rotation networks. At regular checkpoints we record three conflict signs: the minibatch main-task gradient against the auxiliary gradient, the shard reference against the auxiliary gradient, and an oracle direction against the auxiliary gradient. The oracle is the gradient of main-task loss on a large held-out labeled set. It is used only for analysis, never for training.

If minibatch conflict already tracks oracle conflict well at 4 labels per class, the proposal's premise is wrong. In that case the work reduces to a plain asymmetric-PCGrad study. We would report it as such.

The second stage builds the method and its ablations: number of shards, refresh interval, agreement threshold, and cosine-gate versus coordinate-sign variants. Reference estimation adds a full pass over the labeled set every refresh. At 1% ImageNet this pass covers about 13k images, so overhead is small relative to the unlabeled stream. We will report wall-clock cost alongside accuracy.

The third stage tests generality. We repeat the experiments on few-shot base training on MiniImageNet with a rotation auxiliary, and on ImageNet 1% in the S4L-Rotation setup.

Baselines

The naive reference point is supervised training on the labeled subset alone. It uses the same backbone, augmentation, and step count. It tells us whether any auxiliary signal helps at all.

The strongest existing competitor is tuned fixed-weight joint training, exactly as in S4L-Rotation. On ImageNet with 1% labels, S4L reports 53.37% top-5 accuracy for S4L-Rotation. With 10% labels, its full multi-stage MOAM pipeline reports 91.23% top-5. Our single-stage method is not expected to match a pipeline that adds adversarial training, entropy minimization, and pseudo-label retraining, so MOAM is reported as context rather than as the target.

We will reproduce S4L-Rotation in our codebase and compare against our reproduction, not the quoted number. Its loss weight will be tuned on a validation set of the same size we allow ourselves, one or a few labels per class. The search uses the same total GPU budget our method consumes across its own ablation sweep. Beating it means higher top-5 at 1% labels than the reproduced S4L-Rotation by more than the seed-to-seed standard deviation over three seeds, with our method run at a single default threshold.

The gradient-manipulation baselines are run on identical data and steps. They are PCGrad, GradNorm, and TGP-style projection of the primary gradient. They also include AMGS-style discard gating transplanted to the task level, meaning the auxiliary update is dropped when its cosine with the minibatch primary gradient is negative.

The ablation that isolates the key ingredient is minibatch asymmetric projection. It uses the same rule as ours but sets $g_{\text{ref}}$ to the current minibatch main-task gradient. If it matches our method, the cross-shard reference contributes nothing and the claim fails.

An oracle upper bound uses the large held-out labeled gradient as reference. It shows how much of the achievable gain the shard estimate recovers.

On MiniImageNet, the reference figure from the file is TACO at 61.13% 5-shot accuracy versus ProtoNet's 54.80%. For the self-supervised few-shot paper, no reported number was found on Lacuna, so we will rerun it.

Why not just…?

Why not just pre-train with self-supervision and then fine-tune on the labels? This is the most serious threat, because it sidesteps balancing entirely. The file offers indirect evidence that joint training helps. AMGS keeps its masked-token auxiliary active during target adaptation to stop a pretrained model memorizing a tiny target set. Situ et al. report gains from concurrent over sequential auxiliary training. Neither result is decisive for vision at 1% labels. We will therefore include a rotation-pretrain-then-finetune baseline with matched total steps. If it ties our method, we will say joint training is not worth its complexity in that regime.

Why not just grid-search a fixed weight? It can work, and S4L shows tuning on tiny validation sets is feasible. The question is whether that search is reliable and cheap. GradNorm reports that adaptive balancing in one run beat extensive grid search. TGP-MTL found unprojected joint training erratic across folds. Our compute-matched comparison against tuned S4L-Rotation settles this directly. We will also report the across-seed variance of the selected weight.

Why not just use asymmetric PCGrad on minibatch gradients? This is the obvious version of the idea and the one closest to existing work. It is our central ablation. The diagnostic stage predicts it fails at very low label counts, because its conflict test is noise. If it does not fail, our contribution shrinks to that simpler method, and we will report it that way.

Decisive Checks

The proposal stands or falls on three measurements. First, at 4 labels per class on CIFAR-100, oracle conflict must agree more often with shard-reference conflict than with minibatch conflict. Second, our method must beat the minibatch-reference ablation by more than seed noise at that label count. Third, the gap between the two should shrink as labels increase to 100 per class, as expected when minibatch noise falls.

A gain that appears at 100 labels but not at 4 would contradict the mechanism, even if the headline accuracy improved. Holding up at ImageNet 1% against the reproduced S4L-Rotation number, with comparable wall-clock cost, is the final check.