When one neural network is trained on several tasks at once, the tasks often pull its shared weights in opposite directions. The usual fixes either edit the gradients at every step, which requires a separate backward pass per task, or add task-specific modules that can change feature magnitudes as freely as they change directions. We propose a structural middle path. Each task sees the shared weight matrix of a layer through its own rotation. The rotation is built as a short chain of Householder reflections, and a Householder reflection is a mirror flip across a hyperplane, specified by a single learnable vector. Any product of reflections is exactly orthogonal. It can re-orient what a task reads from a shared layer, but it cannot rescale it, and the gradient that each task sends back to the shared weights keeps its size and changes only its direction.
The payoff is diagnostic as well as practical. Negative transfer, where joint training does worse than separate models, is usually blamed on two failure modes at once: one task's large gradients drown out the others, and task gradients point in conflicting directions. An exactly norm-preserving task transform acts only on the second mode. Attaching rotations at controlled depths therefore lets us measure how much negative transfer is directional and where in the network it arises. That question the current literature answers only at the last shared layer.
Core Claim
For a shared linear layer with weight $W$, task $k$ uses $W H_k$, where $H_k = \prod_{i=1}^{r}(I - 2u_{k,i}u_{k,i}^\top/\|u_{k,i}\|^2)$ is a product of $r$ reflections initialized near the identity, so training starts as ordinary hard sharing. Suppose $G_k$ is the gradient of task $k$'s loss with respect to its effective weight $W H_k$. Then the update to the shared $W$ is
$$\nabla_W \mathcal{L} = \sum_{k=1}^{K} G_k H_k^\top, \qquad \|G_k H_k^\top\|_F = \|G_k\|_F .$$
The first expression says the summed update is built from each task's gradient expressed in that task's own rotated frame. Conflicting tasks can therefore learn to write into different directions of $W$'s input space rather than cancelling. The second says this re-orientation leaves each task's gradient magnitude unchanged, because orthogonal matrices preserve the Frobenius norm. Our claim is twofold. First, per-task reflection chains placed inside the backbone, not only before the task heads, reduce negative transfer at a cost of $O(Kr d)$ parameters per layer, where $d$ is the layer width. Second, because they cannot touch gradient magnitude, pairing them with a separate loss-weighting scheme cleanly separates the directional and magnitude contributions to negative transfer.
Relation to Prior Work
The closest work is RotoGrad (also here; RotoGrad: Gradient Homogenization in Multi-Task Learning). It inserts a full task-specific rotation between the last shared representation and each head, trains it with a slower learning rate as a leader-follower game, and reweights tasks by convergence. RotoGrad motivates this proposal: it shows that rotations reduce conflict. It also leaves two gaps that we target. It rotates only the final representation, and its full rotations cost $O(Kd^3)$, with structured parameterizations explicitly left open.
Our proposal differs in three ways. It rotates weights at intermediate layers, not activations after the backbone. It uses cheap reflection chains. And it treats depth of placement as the object of study.
MOORE is also related. It orthogonalizes the outputs of a mixture of experts with a Gram-Schmidt layer in multi-task reinforcement learning. Its constraint sits on representations, not weights, and Gram-Schmidt can become unstable when inputs are nearly collinear. Reflection chains have no such singularity.
Two papers supply the method. oRNN showed that products of reflections give exact orthogonality at $O(nm)$ cost. HRA used reflection chains as a single-task adapter for frozen pretrained weights. It proved that such chains are equivalent to an adaptive low-rank update, and it exposed a regularizer that trades strict orthogonality against capacity. Both works are single-task. HRA's capacity-constraint trade-off becomes, in our setting, a testable question: does multi-task benefit come from the orthogonality or from the added low-rank capacity?
Rethinking Hard-Parameter Sharing constrains our design and also gives us a prediction. It found bottom layers to be the ones most sensitive to task interference, and it made about 20% of bottom-layer filters task-specific by hand. If that finding holds, rotations should help most at the bottom rather than at RotoGrad's top-layer placement.
Among differentiable sharing methods, TAPS gates whole layers between frozen and task-private weights, and EMTAL routes over low-rank experts and then merges them. Neither constrains the geometry of what tasks share.
Research Program
We will build reflection-chain task transforms for convolutional layers, applied to the input-channel dimension, and for transformer linear layers. We will then study three things.
The first is placement. We will attach chains to the bottom, middle, or top third of the backbone at a matched total reflection budget. An honest cost follows from this choice. Once activations pass through a task-specific transform, every layer above it must be computed once per task. Bottom placement therefore multiplies the forward cost by roughly $K$, while top placement is nearly free. We will report wall-clock time and FLOPs alongside accuracy. We will also compare against gradient-surgery methods, which pay a $K$-fold cost in backward passes instead.
The second is the training signal for the reflection vectors. In one variant, the vectors are trained on the summed task loss with a single backward pass. In the other, a RotoGrad-style auxiliary term rewards alignment of task gradients, measured only at the rotated layers.
The third is decomposition. We will measure the cosine between task gradients on each shared layer throughout training, with and without rotations, and cross rotations with and without magnitude-balancing loss weights. This two-by-two design attributes each task's change in the relative drop to directional versus magnitude effects.
Baselines
The naive reference points are single-task models, one per task with the same backbone, and plain hard sharing with equal loss weights. Every method's relative drop $\Delta m\%$ is computed against the single-task models, and hard sharing is the floor any added structure must clear.
The strongest existing gradient-space method is CAGrad. At each step it solves a small constrained problem that maximizes worst-case task improvement within a ball around the average gradient. On NYUv2 it reports a per-task average drop of 0.20%, against 3.97% for PCGrad and 5.59% for MTAN. Beating it means a lower $\Delta m\%$ under the same backbone, epochs, and augmentation, or matching it at lower training cost. GradOPS, which uses Gram-Schmidt projection over all other task gradients, is run as a second gradient baseline. No absolute NYUv2 number for GradOPS was found in the research file.
The strongest structural competitor is RotoGrad. It reports a surface-normal median angle of 20.80 on NYUv2, against 26.16 for vanilla multi-task training. We will run it in its published last-layer configuration and also with reflection chains at the last layer. That second configuration isolates depth from parameterization.
All methods get the same tuning budget for learning rate and method-specific knobs. These knobs are $c$ for CAGrad, the leader learning rate for RotoGrad, and $r$ and placement for ours. We use three seeds each.
The key ablations target the constraint itself. The first replaces each reflection chain with an unconstrained task-specific low-rank update of the same rank and parameter count, in the style of LoRA. The second relaxes orthogonality through HRA's regularizer set to zero. If either ablation matches the orthogonal version, the claim that norm preservation matters fails. A third ablation adds the bottom-specific filters with separate batch normalization from Rethinking Hard-Parameter Sharing at an equal parameter budget. That paper reports 69.2% on CUB-200-2011 against 66.7% for task grouping.
Why not just…?
Why not just use gradient surgery like CAGrad? CAGrad's 0.20% $\Delta m$ on NYUv2 is close to zero, which leaves little headroom. It is a real threat on three-task problems. Its weakness is cost: it needs $K$ task gradients per step. That cost matters more as $K$ grows, as on CelebA with 40 tasks. There, top-placed rotations need only a single backward pass. If CAGrad wins on NYUv2 and we fail to win on cost or quality at large $K$, the proposal is negative. We will report that result.
Why not just give each task a private low-rank adapter? Under HRA's equivalence, a reflection chain is itself a structured low-rank update, so the gain could simply be extra per-task capacity. The matched-rank unconstrained ablation settles this. The orthogonality claim survives only if the constrained version reduces measured gradient conflict and $\Delta m$ more than the unconstrained one at equal parameters.
Why not just rotate the last layer, as RotoGrad does, with a cheaper parameterization? This is the simplest extension and may capture most of the benefit. The placement study compares it directly with deeper placement. The bottom-layer sensitivity reported in Rethinking Hard-Parameter Sharing is the reason to expect depth to matter. The $K$-fold forward cost of bottom placement is the reason it might not be worth it.
Why not just tune loss weights? Loss weights act only on magnitude. The two-by-two decomposition measures how much negative transfer remains after magnitude is balanced. If little remains, directional methods, including ours, have limited value on these benchmarks.
Proposed Experiments
The primary benchmark is NYUv2, with semantic segmentation, depth, and surface normals. We report per-task metrics and $\Delta m\%$ against single-task models, with comparisons to the CAGrad and RotoGrad figures above. For many tasks we use CelebA with 40 attributes, reporting average F1. RotoGrad reports the highest average F1 there, but the research file gives no reported numbers, so all baselines will be rerun. For a controlled test we use Multi-MNIST, where interference between the two digits is synthetic. CosReg reports a 92.5% harmonic-mean accuracy on the related MultiDigitMNIST. Multi-MNIST lets us check whether rotations learn to separate digit-specific input subspaces, which we will inspect through the principal angles between the $H_k$. As a stretch, Meta-World MT10 tests transfer to reinforcement learning against MOORE's reported 88.7% success rate.
Three outcomes are decisive. The first is whether the orthogonal chains beat the matched unconstrained low-rank ablation on NYUv2 $\Delta m$ with non-overlapping seed spread. The second is whether gradient cosines between tasks at rotated layers rise measurably, meaning they become less negative, compared with hard sharing. The third is whether the best placement is bottom rather than top, as predicted by the bottom-layer sensitivity finding. If orthogonality gives no advantage over capacity and placement does not matter, we will conclude that per-task geometric constraints add nothing beyond what RotoGrad and simple adapters already provide.
References
- RotoGrad: Gradient Homogenization in Multitask Learning
- RotoGrad: Gradient Homogenization in Multi-Task Learning
- Multi-Task Reinforcement Learning with Mixture of Orthogonal Experts
- Efficient Orthogonal Parametrisation of Recurrent Neural Networks Using Householder Reflections
- Bridging The Gap between Low-rank and Orthogonal Adaptation via Householder Reflection Adaptation
- Rethinking Hard-Parameter Sharing in Multi-Task Learning
- Task Adaptive Parameter Sharing for Multi-Task Learning
- Transforming Vision Transformer: Towards Efficient Multi-Task Asynchronous Learner
- Conflict-Averse Gradient Descent for Multi-task learning
- Gradient Deconfliction via Orthogonal Projections onto Subspaces For Multi-task Learning