Selecting Auxiliary Tasks by How Forgiving Their Loss Weight Is

Adding auxiliary tasks means training a network on extra objectives, such as predicting depth or a computed molecular descriptor, so that the main task generalizes better. In practice this is notoriously finicky. A task that helps at one loss weight hurts at another, and the methods that balance loss weights automatically bring their own knobs, which also need tuning. Current selection methods ask a yes/no question: does task $k$ help? They answer it at a single weight, usually 1, or with a local gradient test at whatever weight training happens to be using. We argue the useful question is different. Over how wide a range of loss weights does task $k$ help the main task? A task that helps only inside a narrow band of weights is the kind of task that makes training sensitive to hyperparameters. A task that helps across orders of magnitude of weight is safe to include and easy to balance.

The proposal has two parts. The first is a cheap way to measure each candidate's weight-response curve: how much the main task's validation loss improves as a function of that task's weight. It uses short branched training runs from a shared checkpoint. The second is a selection rule that keeps tasks according to the width of their beneficial weight range rather than their peak benefit. The testable consequence links the two research directions. If narrow-band tasks are what make loss-balancing methods fragile, then pruning them should make methods like GradNorm and CAGrad noticeably less sensitive to their own meta-hyperparameters, and it should match or improve main-task accuracy at a fraction of the tuning cost.

Core Claim

For candidate task $k$ and checkpoint $\theta_t$, let $\theta^{(w)}_{t+\Delta}$ be the parameters after $\Delta$ steps of training on the main loss plus $w$ times task $k$'s loss. Define the response curve and its beneficial width as

$$ R_k(w) = L_{\text{val}}\big(\theta^{(0)}_{t+\Delta}\big) - L_{\text{val}}\big(\theta^{(w)}_{t+\Delta}\big), \qquad W_k = \int \mathbf{1}\big[R_k(e^{u}) > \epsilon\big]\, du , $$

where $L_{\text{val}}$ is the main-task validation loss, $u$ is log-weight, and $\epsilon$ is a noise threshold set from the seed-to-seed variance of the $w=0$ branch. $W_k$ is measured in log-weight, so it does not change if a loss is rescaled by a constant. A task that "hurts" only because its loss is numerically 100 times larger keeps the same width; its curve simply shifts. Our claim is threefold. First, $W_k$ predicts a task's eventual contribution under realistic tuning budgets better than peak benefit, instantaneous gradient cosine, or first-order lookahead slope. Second, much reported negative transfer is a weight-scale artifact that shows up as a shifted curve rather than a curve with no positive region. Third, pools pruned by width make downstream balancers flatter with respect to their meta-hyperparameters.

Relation to Prior Work

Several methods already adjust auxiliary weights during training, and each one looks at a single point on the weight axis. The gradient-similarity rule of Adapting Auxiliary Losses Using Gradient Similarity gates or scales an auxiliary gradient by its cosine with the main gradient. Adaptive Auxiliary Task Weighting for Reinforcement Learning (OL-AUX) follows a first-order, multi-step estimate of how a weight change affects the main loss. Auxiliary Learning as an Asymmetric Bargaining Game (AuxiNash) tunes bargaining powers with implicit differentiation on a validation split. All three estimate a local derivative at the current weight. None of them asks whether benefit survives a weight miscalibration. They therefore supply the local signals that we use as competing predictors, but they do not supply the selection criterion.

ForkMerge is the closest work. It forks the model into main-only and auxiliary branches, trains each for a fixed interval, and searches a parameter-interpolation coefficient on validation data. We take two things from it. One is its central empirical lesson: gradient conflict does not imply harm, since L2 weight decay conflicts with the main gradient yet helps generalization. This constrains us to validation-based rather than gradient-based measurement. The other is its fork-and-evaluate mechanism, which we reuse. We differ in three ways. We probe the loss-weight axis directly, at several log-spaced weights, rather than a single interpolation coefficient between branches trained at a fixed weight. We probe one candidate at a time, so memory stays at two models instead of $K+1$. And we select on the width of the beneficial range, which ForkMerge does not measure.

Static selectors make a decision before training and never revisit the weight. GradTS ranks tasks by attention-head gradient correlation and then trains with fixed weights. The conditional-independence filters of Pretext Tasks selection for multitask self-supervised speech representation learning work the same way. AUTAUT pairs an LLM-ranked top-5 list of RDKit descriptors with alignment-based weighting. These methods motivate the gap. Their selection is made at an implicit weight and then handed to a balancer whose sensitivity is never examined.

The balancers come from the multi-task literature: GradNorm, PCGrad, CAGrad and LDC-MTL. They are symmetric and introduce their own meta-hyperparameters, such as GradNorm's asymmetry parameter and CAGrad's conflict radius. They are the objects whose sensitivity we measure, not methods we replace. Finally, Continual Auxiliary Task Learning observes that auxiliary utility decays as tasks are mastered. This is why we re-probe periodically instead of selecting once.

Research Program

The probe works as follows. At a checkpoint, for each candidate task we run $m$ short branches (we expect $m=6$, log-spaced weights spanning four orders of magnitude around a gradient-norm-matched reference) plus one main-only branch. Every branch uses the same minibatch sequence so that differences reflect the weight rather than data noise. The noise threshold $\epsilon$ comes from two reseeded main-only branches. The probe costs $K \cdot m \cdot \Delta$ steps per round with only two models in memory at once, and it parallelizes trivially across devices.

Selection and scheduling build on the probe. The task pool keeps candidates whose $W_k$ exceeds a threshold, and each kept task is assigned the log-weight at the center of its beneficial interval. Training then proceeds either with these fixed weights or with any balancer applied to the pruned pool. Probes are repeated a few times during training. A task whose width collapses is dropped, which handles the temporal obsolescence described in prior work. Total probe compute is reported, and every baseline is given the same extra budget.

The analysis has three parts. We fit a full-training "oracle" weight curve for a subset of tasks using one-dimensional sweeps of complete runs. We then measure how well short-horizon $W_k$, peak $R_k$, lookahead slope and gradient cosine rank tasks against the oracle. Short-horizon probes are known to be myopic, so we test several values of $\Delta$ and report where the rank correlation breaks down.

Baselines

The naive reference points are single-task training on the main objective and uniform weighting of all candidates at $w=1$. ForkMerge reports that uniform weighting falls 10–15% below single-task learning on DomainNet, which establishes that the uniform reference can be badly harmful. Beating single-task training is the minimum bar. Beating uniform weighting is expected and does not count as evidence on its own.

The strongest existing method is ForkMerge. It is validation-based, already prunes tasks by driving their coefficients toward zero, and reports a 4.03% relative improvement over single-task learning on NYUv2, a +2.00% gain on DomainNet, and a 1.30% gain on AliExpress. AuxiNash is the strongest asymmetric gradient-level competitor and reports a −6.80% relative performance drop on NYUv2 and 52.52 mIoU on Cityscapes. Both will be rerun with their released settings on our pools, given the same total compute, including our probe budget, and the same validation split. To beat them, our method must match or exceed ForkMerge's main-task metric at equal compute while using less memory. Separately, it must show a smaller performance spread across its own hyperparameters, here the probe horizon and width threshold, than ForkMerge shows across its fork interval.

The local-signal baselines are cosine gating and an OL-AUX-style lookahead slope. Each is used both as an online weighting method and as a selection score. The symmetric balancers GradNorm, PCGrad and CAGrad are run on the full pool and on our pruned pool. On MoleculeNet we add AUTAUT, which reports gains of +6.4% on BBBP and +5.1% on BACE over baselines.

The expensive reference is a per-task one-dimensional weight sweep using full training runs. It is the oracle our cheap probe tries to approximate, and it is reported with its true cost. Ablations isolate the key ingredient in three ways: selecting by peak benefit instead of width using the same probes, using a single-weight probe at the reference weight, and making a one-time selection versus re-probing.

Why not just…?

Why not just tune each task's weight with a one-dimensional grid of full runs? This is the real threat, and it is the oracle baseline. It costs $K \cdot S$ full trainings for $S$ grid points, rather than the $S^K$ cost of a joint grid. If our short probes do not rank tasks the way the oracle does, the method has no reason to exist, although the measurement claim that harm is often a curve shift could still stand. The decisive comparison is rank correlation with the oracle and final accuracy per unit of compute.

Why not just use ForkMerge, which already validates against the main task? ForkMerge measures whether a branch trained at a fixed weight is worth interpolating toward. It does not measure how sensitive that judgment is to the weight, and it keeps $K+1$ models in memory. If ForkMerge with a tuned fork interval matches our accuracy and sensitivity at equal compute, our contribution shrinks to a memory saving, and we will report that outcome.

Why not just run GradNorm or CAGrad on all candidates and let the balancer sort it out? These balancers are symmetric, and their meta-hyperparameters are themselves the sensitivity problem. AuxiNash also argues that symmetric balancing sacrifices main-task progress to protect auxiliary objectives. Our sensitivity experiment tests this directly. If balancer spread across its own meta-hyperparameter grid does not shrink on the pruned pool, the central hypothesis connecting the two directions is false.

Why not just select by gradient cosine? ForkMerge's weight-decay result shows that conflicting gradients can be helpful. The rank-correlation analysis will show whether cosine predicts oracle benefit at all once tasks are rescaled.

Proposed Experiments

We use four testbeds. The first is NYUv2 with segmentation as the main task, depth and normals as auxiliaries, plus added self-supervised candidates such as rotation prediction and image reconstruction. Rotation is a deliberately risky candidate given the orientation dependence reported in Self-Supervised GANs via Auxiliary Rotation Loss. The second is Cityscapes in the AuxiNash configuration. The third is DomainNet, which has strong documented negative transfer. The fourth is MoleculeNet BBBP and BACE with a pool of roughly twenty to thirty RDKit descriptors as candidates. The molecular pool is large enough that selection matters, and it avoids LLM ranking.

There are three decisive checks. The first is a scale-artifact test: we duplicate each candidate with its loss multiplied by 10 and by 100. Width-based selection should keep or drop the copies together with the originals, while fixed-weight and cosine-based selection should flip. The second is a sensitivity test: for GradNorm and CAGrad we sweep their meta-hyperparameter and learning rate on the full and pruned pools, and report best-minus-worst spread and expected accuracy under a uniformly random draw from the grid. The claim fails if the pruned pool does not reduce the spread. The third is a predictive test: Spearman correlation between short-probe $W_k$ and oracle benefit across tasks and seeds, compared against peak $R_k$, lookahead slope and cosine.

All results are averaged over at least three seeds and reported with probe compute included. Reported reference values that we cannot reproduce exactly, such as CAGrad's 0.20% $\Delta m$ on NYUv2 in the symmetric multi-task setting, are cited only as context because their task protocol differs from our main-task-only metric.