When one network predicts several things about an image, some predictions change with the camera and some do not. Suppose you roll the camera, rotating the image in its own plane. The semantic label at each pixel stays the same once you follow the pixel to its new position, and so does the depth. The surface normal, the direction a surface faces, is different. Its sideways components turn with the camera. A shared backbone trained on all three tasks therefore has to encode two kinds of features in the same channels: features that stay fixed under the rotation and features that turn with it. We propose to split the shared latent explicitly. One block is regularized to be invariant, and it feeds segmentation and depth. The other block, which we call the fiber, is regularized to rotate exactly as the normals do, and it feeds the normal head. We then test whether this split removes the gradient conflict that multi-task methods currently handle with rotations or projections applied after the fact.
This matters because the literature has settled on a common diagnosis: tasks fight over shared parameters, and their gradients point in opposing directions. Surface normals are the usual casualty on NYUv2. Prior work has treated that conflict as a statistical fact to be managed. If a large part of it instead comes from a mismatch between how each task's target transforms, then a cheap, known geometric prior should prevent the conflict from arising at all. That is a more specific and more falsifiable claim than "geometry helps."
Core Claim
The claim has two parts. First, some of the conflict between normal estimation and the invariant tasks comes from asking the same latent channels to be both invariant and equivariant to in-plane rotation. Equivariant here means that the features transform in a prescribed way when the input transforms. Second, giving the backbone a latent split into an invariant base and an SO(2) fiber reduces how often task gradients conflict in the last shared layers. The fiber is made of channel pairs that rotate like 2-D vectors, and the split is enforced with a soft regularizer on self-generated rotated pairs. Fewer conflicts should in turn improve normals without hurting segmentation or depth. The new ingredient is not the rotated pairs, since augmentation already produces them. It is the constraint on how the representation must respond to them, fitted to each task's transformation behaviour.
Relation to Prior Work
The closest work is RotoGrad. It inserts a learned task-specific rotation in SO(d) between the final shared feature and each head, and optimizes those rotations to align task gradients. It reports a normal median angle error of 20.80° on NYUv2, against 26.16° for plain multi-task training. RotoGrad's rotations are abstract coordinate changes learned purely from gradient statistics, and they bear no relation to any physical transformation. Our proposal does the reverse: it fixes the group, SO(2) acting on the image plane, fixes how each task must respond to it, and leaves the gradient dynamics alone. RotoGrad therefore motivates the proposal, because it shows that normals are the main source of conflict. It is also the method to beat.
Two optimization lines constrain the design. Gradient Surgery for Multi-Task Learning (PCGrad) projects away conflicting gradient components. Recon shows that such projection does not reduce how often conflicts occur. Recon instead branches the most conflicted layers, which it finds with a layer-wise "S-conflict score." We borrow that score as our main diagnostic. Our prediction is that the symmetry split lowers it in the final shared layers without adding parameters and without Recon's separate pilot training phase.
The geometric side supplies the method. Structuring Representations Using Group Invariants regularizes latents toward group structure using pairs of an input and its transformed copy, rather than using a hard-wired equivariant architecture. It was used only for world models and reinforcement learning. Isometric Quotient VAEs and Manifold Learning in Quotient Spaces split representations into a part that ignores a symmetry and a part that carries it. The quotient space is the part that ignores it, and the group is the part that carries it. Both were used only in single-task autoencoders. Their gains over augmentation-trained baselines, an adjusted Rand index (ARI) of 70.9 against 11.4 on rotated MNIST and up to 20% lower reconstruction error on ShapeNet, are the evidence that structuring the latent can beat merely showing the model rotated data. We carry that decomposition into supervised multi-task heads.
Disentangled Representation Learning with the Gromov-Monge Gap warns that geometric penalties which demand an impossible exact structure produce unstable gradients. We take this warning seriously. Our constraint is exactly satisfiable, because the action is known and the representation is chosen to match it, and we keep it soft.
Pattern-Affinitive Propagation couples depth, normals and segmentation through pixel-to-pixel affinity diffusion on the feature map. It is complementary to our approach. It does not model how targets transform, and it is a heavier architecture, so it serves as a reference point rather than a like-for-like competitor.
Research Program
We use a standard multi-task dense-prediction backbone with three heads on NYUv2. The last shared feature map has $C$ channels per pixel, which we split into an invariant block $a$ and a fiber block $u$. The fiber consists of $m$ channel pairs, and each pair is treated as a 2-D vector.
The heads are wired according to how each target transforms under camera roll. The segmentation and depth heads read $a$, together with the per-pair norms of $u$, since those norms are invariant. The normal head predicts the out-of-plane component $n_z$ from $a$, and it predicts the in-plane components $(n_x, n_y)$ as a learned linear combination of the fiber pairs. By construction, this combination rotates with the camera.
For each training image we draw an angle $\theta$, form the rotated image $R_\theta x$, and add the following regularizer:
$$\mathcal{L}_{\text{sym}} = \mathbb{E}_{x,\theta}\Big[\big\|a(R_\theta x) - W_\theta\, a(x)\big\|^2 + \big\|u(R_\theta x) - \rho(\theta)\, W_\theta\, u(x)\big\|^2\Big]$$
Here $W_\theta$ spatially rotates a feature map by $\theta$. The term $\rho(\theta)$ rotates every fiber channel pair by $\theta$. The norms are computed only over pixels that are valid in both views, which excludes the black corners introduced by rotation. The total loss is the sum of the task losses and $\lambda \mathcal{L}_{\text{sym}}$. The rotated view is also supervised with correctly transformed labels: segmentation and depth maps are warped, and normal vectors are both warped and rotated. Horizontal flips are handled the same way, with the flip acting on the fiber as a reflection.
Collapse of the fiber to zero would satisfy the regularizer trivially. The normal head's dependence on the fiber resists this. We will also monitor fiber norms, and if collapse appears we will add the log-barrier from the group-invariants paper.
A controlled testbed comes first, because NYUv2 contains little natural camera roll. We build a two-task rotated-MNIST problem, following RotoGrad's use of Multi-MNIST: digit class is the invariant task, and orientation regressed as a unit vector is the equivariant task. This setting has exact ground-truth symmetry, and it shows whether the mechanism works at all.
We then move to NYUv2 with 13-class segmentation, depth and normals. We report mean IoU, depth RMSE and absolute relative error, normal mean and median angle error with within-11.25° accuracy, and the average relative improvement over single-task networks used by Recon. Every result is reported on both the standard test split and a rotated test split. Throughout, we record layer-wise S-conflict scores between each pair of tasks.
Baselines
The naive reference is plain multi-task training with hard parameter sharing, which RotoGrad reports at a 26.16° normal median angle error. Next to it we run the same model with rotation and flip augmentation and correctly transformed labels. This baseline sees exactly the same rotated pairs, the same number of forward passes and the same label supervision as our method, and it lacks only $\mathcal{L}_{\text{sym}}$ and the split heads. It is the most important baseline, because any gain that augmentation alone can explain is not ours.
The strongest existing method on this exact question is RotoGrad, at 20.80° median angle on NYUv2. It also targets normal conflicts through geometry at the feature-to-head interface. We rerun it in our backbone with the same augmentation, rather than copying the reported number, because backbones differ.
We also run Recon on top of CAGrad, the configuration it reports as moving NYUv2 average relative improvement from 3.74% to 5.80%. We choose it because it attacks conflict frequency directly, which is the same quantity our diagnostic measures. PCGrad is included as the standard gradient-surgery reference.
All methods receive the same training schedule and augmentation. Each gets an equal hyperparameter budget of a fixed number of runs, spent on $\lambda$ for ours, the rotation learning rate for RotoGrad, and $K$ and $S$ for Recon. All results use three seeds.
Two ablations isolate the key ingredient. The first uses split heads with the same channel routing but no $\lambda \mathcal{L}_{\text{sym}}$, which tests whether routing alone does the work. The second keeps $\lambda \mathcal{L}_{\text{sym}}$ but makes the whole latent invariant with no fiber, which tests whether the equivariant block matters or whether invariance is simply a regularizer.
Beating the field means two things. Normal median error must fall below both augmented plain multi-task training and our rerun of RotoGrad by more than the seed spread, while segmentation and depth do not degrade. Separately, the last-layer S-conflict score must fall relative to augmentation-only training. Stacking our method with CAGrad or RotoGrad is reported separately, to show whether the effects are complementary.
Why not just…?
Why not just augment with rotations and transform the labels correctly? This is the real threat. A network trained this way might discover the invariant/fiber split by itself. The single-task evidence from Isometric Quotient VAEs and Manifold Learning in Quotient Spaces says it usually does not. There, augmentation-trained models lagged far behind structured latents. That evidence, however, comes from unsupervised settings. The augmentation-matched baseline settles the question directly: if it matches us on task metrics and on conflict scores, the proposal's thesis is wrong for supervised multi-task learning, and we will report it that way.
Why not just use RotoGrad or gradient surgery? These methods reshape gradients once conflict has arisen, and Recon shows that projection-style fixes leave the frequency of conflicts unchanged. RotoGrad's learned rotations could approximate a symmetry split if the data supported it. Our conflict-score comparison and head-to-head rerun test whether a known physical constraint does better than a learned abstract rotation. If they tie, the combination experiment shows whether the two are redundant.
Why not just branch the conflicting layers, as Recon does? Branching removes conflict by giving up sharing. It adds a pilot phase, discrete hyperparameters and extra parameters. Our method keeps a single shared backbone at the same parameter count. If Recon wins on task metrics, we will say so. The comparison that matters is performance at equal parameter count, and the conflict scores show whether ours achieves Recon-like reductions without splitting.
Why not just use a hard-wired rotation-equivariant convolutional backbone? Such architectures enforce equivariance exactly but give up standard pretrained backbones, and none appears in our source material with multi-task results. The soft regularizer is a cheaper intervention that works with any backbone. We treat an equivariant-architecture comparison as optional future work, not a claim.
Decisive Checks
The first check runs on the rotated-MNIST testbed. The symmetry split must lower the conflict score between the class and orientation tasks, and it must improve orientation error over augmentation-only training. If it fails here, where the symmetry is exact, we stop.
The second check runs on NYUv2. The normal-versus-segmentation and normal-versus-depth conflict scores in the last shared layers must drop relative to the augmentation-matched baseline. The normal gains must appear on the standard split, not only on rotated test images. A gain on rotated test images alone would mean we had solved a problem we created.
The third check is the pair of ablations. The regularizer must beat routing alone, and the fiber variant must beat the fully invariant variant. Only if all three checks hold can the claim be stated as proposed: symmetry mismatch in the shared latent is a cause of negative transfer, not just a correlate of it.
References
- RotoGrad: Gradient Homogenization in Multitask Learning
- Gradient Surgery for Multi-Task Learning
- Recon: Reducing Conflicting Gradients From the Root For Multi-Task Learning
- Structuring Representations Using Group Invariants
- Isometric Quotient Variational Auto-Encoders for Structure-Preserving Representation Learning
- Manifold Learning in Quotient Spaces
- Disentangled Representation Learning with the Gromov-Monge Gap
- Pattern-Affinitive Propagation Across Depth, Surface Normal and Semantic Segmentation