Finding Where Tasks Fight: Self-Expressive Clustering of Task Gradients

When one network is trained on several tasks at once, the tasks can work against each other. One task's gradient pushes the shared features one way and another task's gradient pushes them back. The best-known fixes either reweight whole tasks or rotate the shared feature space for each task. Both act on the full feature vector, or on an arbitrary slice of it. This proposal starts from a simple observation: the gradients each task sends back into the shared representation are not spread over all dimensions. They fall on a small number of low-dimensional subspaces, one or a few per task and per kind of input. Recovering that structure is exactly what subspace clustering was built to do. Its standard tool is self-expression, which writes each point as a sparse combination of other points from the same subspace and then clusters the resulting affinity graph.

The idea is to run self-expressive subspace clustering on per-sample task gradients rather than on data. This finds the specific subspaces where gradients from different tasks overlap and point in opposite directions. The conflict-resolving intervention, a task-specific rotation in the style of RotoGrad, is then applied only inside those subspaces. If it works, the method replaces RotoGrad's hand-picked "rotate 1024 dimensions" with a subspace and dimension chosen by the data, and it gives a direct, inspectable map of where negative transfer happens.

Core Claim

Consider a shared feature $z_i \in \mathbb{R}^d$ for input $i$ and task losses $L_k$. Define the per-sample task gradient $g_{k,i} = \partial L_k / \partial z_i$. For a linear head $W_k \in \mathbb{R}^{c_k \times d}$, each $g_{k,i}$ lies exactly in the row space of $W_k$. That is a subspace of dimension at most $c_k$, so a batch of such gradients from many tasks is a union of subspaces by construction. For nonlinear heads such as dense-prediction decoders, we hypothesize that the gradients remain close to a union of low-dimensional subspaces. Under that hypothesis, the subspaces are indexed by task and by input type (for example, a scene region or class), not by task alone.

Our claim is that self-expressive clustering of the pooled gradient set $\{g_{k,i}\}$ recovers these subspaces. Clusters that mix gradients from two or more tasks with negative inner products are "conflict subspaces." Restricting gradient-direction homogenization to these subspaces should match or beat full-space and fixed-slice rotation at a fraction of the rotated dimension. It should also beat single-subspace alternatives such as principal components of the gradients, because those directions capture gradient energy rather than gradient conflict.

Relation to Prior Work

The closest bridge is Multi-Task Learning for Subspace Segmentation. It treats each point's sparse reconstruction as one "task" and couples the tasks with sparse Bayesian learning, which cuts Hopkins 155 three-motion error from 7.15% (SSC) to 3.80%. That paper uses multi-task learning as a tool for clustering. We invert the direction and use clustering as a tool for multi-task learning: our points are the gradients of real, heterogeneous tasks, and our output is an intervention on training. Its finding that coupling helps over independent sparse codes is a motivation for us. We will test it again on our setting by comparing independent per-point codes with grouped ones.

The method we modify is RotoGrad. RotoGrad learns task-specific orthogonal rotations $R_k$ of the shared features to align gradient directions, together with magnitude reweighting. It reports that rotating a 1024-dimensional subspace already outperforms its baselines on NYUv2, but that subspace is chosen by slicing dimensions. We keep RotoGrad's rotation and its leader-follower optimization and change only where the rotation acts. Concrete Subspace Learning also isolates subspaces to reduce interference. It does so in parameter space, after training, using masks learned through a two-level optimization, so it addresses model merging rather than joint training.

On the self-expression side, the solvers come from Sparse Subspace Clustering, the hybrid sparse/low-rank variant of Provable Subspace Clustering: When LRR meets SSC (which improves within-cluster connectivity), and fast solvers. These fast solvers are SSQP, with its reported 50-fold speedup, and SENet, whose learned query-key network assigns new points to subspaces without re-solving and has scaled to more than 190,000 points.

One prior result constrains our design directly. A Critique of Self-Expressive Deep Subspace Clustering proves that jointly learning features and self-expressive coefficients admits collapsed solutions. Without post-processing, accuracy on Extended Yale B drops from 96.79% to 59.09%, below a plain autoencoder at 71.96%. PRO-DSC repairs this with a log-determinant volume term, and DELVE stacks self-expressive layers. We avoid the failure mode entirely: no self-expressive loss ever touches the representation. Self-expression is solved on detached gradient vectors, a fixed point set for which the classical convex guarantees apply. Finally, the LRSC-CKA study shows that self-expressive structure in single-task networks tracks linear-probe accuracy with correlation above 0.9. That is evidence that this geometry is real in trained features, although the study is purely diagnostic and single-task.

Research Program

Every $T$ training steps, we take a probe batch and compute $g_{k,i}$ for all tasks. For dense tasks, we subsample pixels so the point set stays at a few thousand vectors, and we normalize each vector to unit length. We then solve an elastic-net self-expressive program on the pooled set,

$$\min_{C}\ \lambda\|C\|_1 + \tfrac{1-\lambda}{2}\|C\|_F^2 + \tfrac{\gamma}{2}\|G - GC\|_F^2 \quad \text{s.t. } \operatorname{diag}(C)=0,$$

where $G$ stacks the gradient vectors as columns, $C$ holds the reconstruction coefficients, and $\lambda,\gamma$ trade off sparsity, connectivity, and fit. Spectral clustering on $|C|+|C|^\top$ gives clusters, and each cluster's top singular vectors give a basis $U_j$. Cluster dimensions come from a singular-value gap rather than a preset size. A cluster is labeled a conflict subspace if it contains gradients from at least two tasks whose mean projected inner product is negative.

Between refreshes, a SENet-style network trained on the same gradients assigns new gradients to clusters cheaply. RotoGrad's rotations are then parameterized only on the union of conflict subspaces, and the shared features are left untouched in the complement. We also study a cheaper variant that projects out opposing components within conflict subspaces instead of rotating.

The analysis has two parts. The first is a correctness check: on networks with linear heads, the recovered subspaces should coincide with the row spaces of the heads, which we can measure by principal angles. The second is descriptive: how many conflict subspaces appear, how their dimension evolves during training, and whether they align with input groups such as specific semantic classes in NYUv2.

Baselines

The naive reference point is vanilla multi-task training with equal loss weights on a shared backbone, with each task also trained alone to define the relative improvement $\Delta_m$ (average per-task gain over single-task models). This is the number every method must exceed.

The strongest optimization-level competitor is RotoGrad, run in its full-space form and with its 1024-dimensional slice. It is the closest competitor because our method keeps RotoGrad's rotation and changes only where it acts, so any gain is attributable to subspace choice. No reported per-task NYUv2 numbers from RotoGrad were found on Lacuna, so we will reproduce it ourselves. We use the same backbone, data splits, and training schedule as our method, and give every method an equal hyperparameter-search budget measured in GPU-hours.

The strongest architecture-level reference is DeMT. It reports $\Delta_m$ = 3.36% on NYUv2 with a Swin-T backbone, versus 0.31% for MQTransformer, and 46.36 IoU on semantic segmentation. Because our method is an optimization change, we apply it on top of DeMT with Swin-T. Beating this reference means exceeding DeMT's own reproduced $\Delta_m$ when trained with our intervention, not exceeding a different architecture.

Two ablations isolate the key ingredient. The first rotates random subspaces with the same total dimension as our discovered conflict subspaces, which separates "less rotation" from "the right rotation." The second rotates the top principal components of the pooled gradients, again at matched dimension, which separates conflict structure from energy structure. A further ablation replaces the joint self-expressive program with independent per-point sparse codes, which tests whether the connectivity benefit reported by MTSC carries over.

Why not just…?

Why not just reweight the task losses? Loss weighting changes how much each task pulls on the shared features, not the direction of the pull, and RotoGrad's analysis attributes much of negative transfer to direction. Weighting is included through RotoGrad's magnitude term, which we keep in every variant. If a well-tuned weighting baseline matches our method on $\Delta_m$, our method's added value reduces to interpretability, and we will report that plainly.

Why not just take the principal components of the task gradients? This is the most serious threat. Principal components find where gradients are large, and conflict may sit there too. Our argument is that a single principal subspace mixes non-conflicting high-energy directions with conflicting ones, while a union-of-subspaces model separates them. The matched-dimension PCA ablation settles the question directly. We additionally measure what fraction of the total negative inner-product mass between task gradients falls inside each method's chosen subspace. If PCA captures as much conflict mass and gives the same $\Delta_m$, the self-expressive step is unnecessary, and we would recommend PCA.

Why not just read the subspaces off the task heads? For linear heads this is exactly right, and we use it as our correctness check. For decoders, there is no single head row space, and gradients vary with the input region. The test is whether self-expression finds input-dependent subspaces on NYUv2 and PASCAL-Context that per-task subspaces miss, shown as distinct clusters within one task that have different conflict partners.

Why not just put a self-expressive loss on the shared features? The critique paper shows that such losses invite collapse unless the representation is protected, and PRO-DSC's fix adds a volume regularizer that has no tested role in supervised multi-task training. Applying self-expression only to detached gradients avoids this risk, at the cost of not shaping the features themselves.

Proposed Experiments

We start with controlled tests on Multi-MNIST and the CIFAR-10 multi-task setup used by RotoGrad, where heads are linear and the correctness check applies. We then move to CelebA multi-label classification with 40 tasks, which stresses the number of clusters. The main evaluation is NYUv2 with segmentation, depth, and normals, and PASCAL-Context with five tasks, reporting per-task metrics and $\Delta_m$. Both use a ResNet backbone matching RotoGrad and Swin-T on top of DeMT. We also report added wall-clock cost per step, total rotated dimension, and refresh frequency $T$, comparing SSQP-based and SENet-based cluster assignment.

Three outcomes decide the proposal. First, discovered subspaces must recover head row spaces on linear-head tasks with small principal angles. Second, at matched dimension, discovered conflict subspaces must contain more negative inner-product mass than both PCA and random subspaces. Third, on NYUv2, restricting rotation to these subspaces must match or exceed RotoGrad-1024 in $\Delta_m$ while rotating fewer dimensions, and must add to DeMT's gains rather than duplicate them. If the second check fails, the union-of-subspaces hypothesis about gradients is wrong, and the method should be abandoned rather than tuned.