Gate-Protected Alternation for Multi-Task Networks

Multi-task networks save memory and compute by sharing one backbone across several tasks. One cheap way to train them is to alternate: each step computes and backpropagates only one task's loss, picked at random or in turn. This avoids holding every task's computation in GPU memory at once. The same networks increasingly rely on gates, which are small learned multipliers between 0 and 1 that decide how much of each shared feature channel a given task reads. Gates reduce negative transfer, meaning the case where one task's training damages another task's features. The two ideas have not been studied together, and they clash. When task A takes a step, it rewrites shared channels that task B's gates have come to depend on, and B's gates get no signal that this happened. Our proposal is to reuse the gates as a record of which task depends on which channel. On task A's turn, we shrink the update to each shared channel in proportion to how heavily the other tasks currently rely on it. The gates are already computed, so this costs almost nothing.

This matters in practice because alternation is what makes large or heterogeneous multi-task training fit on a device. Mitigating Forgetting in LLM Supervised Fine-Tuning and Preference Learning reports that mixing losses raised GPU memory by over 35%, while alternation added almost no overhead. Alternating Gradient Descent and Mixture-of-Experts for Integrated Multimodal Perception found that alternating across objectives beat summed losses downstream. The strongest dense-prediction models, however, share features through gated cross-task decoders, for example MTMamba. These gated models are always trained with all task losses at once, so it is unknown whether they survive alternation or how they should be adapted to it.

Core Claim

Take a shared layer whose output channels are indexed by $c$. Task $k$ reads channel $c$ through its gate $g_{k,c}(x)\in[0,1]$. We track each task's reliance on channel $c$ as $r_{k,c}$, a running average of $g_{k,c}$ over recent batches. On a step where task $a$ is sampled, the gradient reaching the parameters that produce channel $c$ is rescaled:

$$\tilde{\nabla}_{c} = \Big(1-\beta\,\max_{k\neq a} r_{k,c}\,(1-r_{a,c})\Big)\,\nabla_c L_a ,$$

Here $L_a$ is task $a$'s loss and $\beta\in[0,1]$ is the protection strength. The effect is as follows. Channels that only task $a$ uses get the full update. Channels that other tasks depend on but $a$ barely uses are protected. Channels that everyone relies on are left alone so that positive transfer still flows.

Some protection already happens without this rule. Task $a$'s own gradient into channel $c$ is scaled by $g_{a,c}$, so channels $a$ ignores receive little update. The added term addresses the remaining failure: a channel that $a$ uses moderately but some other task depends on heavily. This is where alternation-induced drift should concentrate. The idea resembles importance-weighted protection in continual learning. The difference is that the importance estimate comes free from the gates rather than from extra Fisher-information passes, and the tasks keep taking turns rather than arriving once in sequence.

The claim can fail in three ways. It fails if plain alternation does not degrade gated models relative to summed-loss training, because then there is nothing to fix. It fails if a uniform learning-rate reduction on shared layers matches the gain. And it fails if stale-gradient projection, discussed below, does as well at similar memory.

Research Program

The first stage is diagnosis. We will train gated multi-task models two ways, with summed losses and with uniform stochastic alternation, and measure what alternation actually does to the gates. The quantities are per-step gate change (jitter), the fraction of gates saturated near 0 or 1, and "idle drift." Idle drift is defined as the change in task $k$'s gated features on a fixed probe batch between two consecutive turns of $k$, compared with the change in $k$'s loss over the same interval. This stage tests the open question raised in the file, which notes that nobody has checked for router destabilization under alternation. It also tells us whether drift is concentrated in heavily-relied-on channels, which is the premise of the rescaling rule.

The second stage is the method. We implement the rescaling rule for two kinds of gates. One is channel-wise sigmoid gates on a shared encoder, as in Liu et al.. The other is the task-specific gates inside MTMamba's cross-task decoder block. For block-level routing as in AdaShare, select-or-skip probabilities play the role of $r_{k,c}$ at block granularity.

Cross-task decoders need every task's features. Under alternation we will compute the idle tasks' decoder features in a forward pass with no stored graph and pass them in as constants. Peak activation memory is then roughly one task's graph plus some cheap buffers. We will report this memory directly.

One secondary component is kept deliberately small. We will choose which task to sample next in proportion to its measured idle drift, and compare this against the scalar loss-lag rule used by MAXRIGHT. We treat it as an ablation, not the core claim.

The third stage tests whether alternation noise removes the need for AdaShare's staged curriculum. Random loss perturbation is reported to find flatter minima (Reasonable Effectiveness of Random Weighting). If that holds for routing policies, alternating updates might keep them from collapsing into degenerate sharing patterns. This stage is exploratory, and a negative result here would not undermine the main claim.

Baselines

The naive reference is a gated network trained with uniform stochastic alternation and no protection, which is $\beta=0$. This is exactly the setting the proposal claims is broken. A second reference is the same gated network trained with the usual summed equal-weight loss, which shows what alternation gives up in exchange for its memory savings. We also report single-task networks, because gains are expressed relative to them.

The strongest existing competitor on accuracy is ForkMerge (ForkMerge). It trains separate branches for a few steps at a time, then merges their weights using a validation-tuned interpolation. On NYUv2 it reports a 4.03% relative improvement over single-task learning and beats CAGrad and NashMTL. It is the strongest competitor because it also works by alternating updates and then reconciling them, only in weight space rather than inside the network. Its weakness on our question is memory, since it stores multiple model copies during each fork.

On loss weighting, the strongest cheap competitor is Random Loss Weighting. The file reports a 1.04% relative improvement over equal weighting on NYUv2 for it (Reasonable Effectiveness of Random Weighting). For the architecture we reuse MTMamba, which reports +2.08 mIoU for segmentation over InvPT on PASCAL-Context. Our question is whether that architecture keeps its accuracy when trained by alternation.

All methods will use the same backbone, the same data, the same number of optimizer steps per task, and the same hyperparameter-search budget. The search covers roughly twenty trials over learning rate and each method's own knob: $\beta$ for ours, the interval and $\lambda$ grid for ForkMerge, and the Dirichlet concentration for RLW. We report peak GPU memory and wall-clock time next to accuracy.

The key ablations are three. The first replaces reliance-weighted scaling with a uniform scaling of shared-layer updates, tuned to the same average magnitude. This separates "use the gates" from "just take smaller shared steps." The second computes reliance from random gates rather than learned ones. The third removes the drift-driven sampler.

Beating the baselines means the following. Under alternation, the method must match or exceed summed-loss training of the same gated model in average relative gain over single-task models. It must close most of the gap between naive alternation and summed training. And it must beat uniform scaling outside the seed variance. Against ForkMerge, the claim is Pareto improvement: comparable accuracy at clearly lower memory. We do not promise to win on accuracy alone.

Why not just…?

Why not just use a hard-shared trunk with Random Loss Weighting? This option adds no parameters and is a legitimate litmus test. However, the file reports that random gradient weighting does poorly on 40-task CelebA. Structural routing in ETR-NLP reaches a higher F-score there with 28.5% fewer learnable parameters. It also gains +3.56 mIoU on Cityscapes. We include RLW on every benchmark. If it matches gated models trained by our method at equal memory, the architectural half of the proposal is unnecessary, and we will report that result.

Why not just apply gradient surgery such as PCGrad or CAGrad? These methods need all task gradients at every step, which is precisely the memory cost alternation avoids. TurboTrain found per-step balancing costly enough to amortize it over "free" steps. The serious version of this objection is a stale-gradient variant: keep each task's most recent shared-layer gradient and project the current task's gradient away from conflicting ones. That variant costs one parameter-sized buffer per task, and with few tasks and small models it could plausibly match us. We will run it as a baseline on NYUv2 and PASCAL-Context. ForkMerge's observation that gradient conflict does not track negative transfer gives some reason to expect a feature-level signal to do better, but the experiment decides.

Why not just keep summed-loss training and use gradient checkpointing to save memory? Checkpointing recovers memory by recomputing activations, so it trades memory for compute. It also leaves heterogeneous input shapes padded to a common size. Alternation avoids padding, as IMP notes. We will report memory and wall-clock time for summed training with checkpointing. If it wins on time at equal memory for dense benchmarks, the practical case for alternation shrinks to heterogeneous-input settings, and we will state that.

Why not just partition channels statically between shared and task-specific branches? A fixed partition, as in ETR-NLP with its sharing ratio $\gamma$, removes drift by construction because idle tasks own their channels. The cost is that the amount of sharing is fixed in advance. Our rule amounts to a soft partition that the gates learn. ETR-NLP trained with alternation is a direct comparison, and we will run it.

Proposed Experiments

Diagnosis and the main comparison use NYUv2 with three tasks (segmentation, depth, normals) and PASCAL-Context with five tasks. They cover two architectures: a gated shared encoder and MTMamba-style gated cross-task decoders. The metric is the average per-task relative gain over single-task models, reported together with peak memory, gate jitter, saturation fraction, and idle drift.

CelebA with 40 tasks is the many-task stress test. It is the regime where random weighting is reported to fail, and where the max-over-other-tasks reliance term will be close to 1 for most channels. That could over-protect the network, so CelebA is where the method is most likely to break. Tiny-Taskonomy tests AdaShare-style block routing, and the curriculum question in particular: AdaShare's policy trained with alternation and no warmup is compared against AdaShare with its curriculum. AdaShare reports +1.1% relative performance with 80% fewer parameters on this benchmark.

The decisive checks are three. First, naive alternation must measurably hurt gated models relative to summed training, with drift concentrated in high-reliance channels. If it does not, the premise fails and we report the diagnostic result by itself. Second, reliance-weighted scaling must beat both uniform scaling and random-gate reliance at matched update magnitude. Third, the method must land on the accuracy–memory Pareto front against ForkMerge, stale-gradient projection, and checkpointed summed training. We will use at least three seeds throughout, because the expected margins are around one to two points of relative gain.