Where Tasks Disagree: Local Task Alignment for Region-Specialized Generative Multi-Task Learning

A multi-task network uses one shared backbone to solve several problems at once, for example predicting depth, surface orientation and object labels from the same indoor photo. It often does worse on some tasks than separate single-task models would. The usual explanation is that the tasks' training signals (their gradients) pull the shared features in conflicting directions. Current fixes treat this conflict as a single property of the whole dataset. RotoGrad, for instance, learns one rotation of the shared features per task and applies it to every image. We propose to test a different picture. Task conflict may be local: depth and surface normals may agree on cluttered bedrooms and disagree on flat, textureless corridors. If so, a single dataset-wide fix averages away the conflict it is trying to remove.

The proposal has three linked parts. First, we measure task agreement only within each sample's close neighborhood in feature space, borrowing the "local kernel alignment" idea from clustering. Local kernel alignment means comparing similarity matrices only between a point and its nearest neighbors, never between distant points. Second, we group the data into regions where tasks relate to one another in the same way. Third, we train one small generator per region. These generators produce synthetic images that carry a region label for free, and those labels drive region-specific feature rotations when real labels are scarce. In short, we split the data according to how tasks relate, not according to what images look like, and we give each such region its own generator and its own conflict fix.

Core Claim

The claim has two parts, and it is falsifiable in stages. The first part is empirical. On standard multi-task benchmarks, how much two tasks agree varies strongly across neighborhoods of the data, and a dataset-wide average hides regions of strong conflict. The second part is methodological. Suppose we define regions by their local task-agreement profile, instead of by pixel modes (as multi-generator GANs do) or by raw feature clusters. Then region-conditional conflict resolution, supported by region-specialized generators, beats a single global rotation in the low-label regime. It should do so at matched parameters and compute.

What would be new is using locality as the unit at which task relationships are measured and corrected, and using that unit to decide what each generator specializes in. Multi-generator methods specialize on visual modes. Multi-task methods correct conflict globally. Local alignment methods have so far been used only for clustering and retrieval.

Relation to Prior Work

The local-alignment direction supplies the measurement technique. Multiple Kernel Clustering with Local Kernel Alignment Maximization aligns kernels only between each sample and its τ nearest neighbors. It reports 96.25% clustering accuracy on the Digital benchmark versus 47% for global alignment. Localized Simple Multiple Kernel K-means and its matrix-induced regularization variant restrict alignment with neighborhood masks and model correlations among the localized kernels. Cluster-Aware Similarity Diffusion for Instance Retrieval restricts affinity to k-reciprocal neighbors, meaning pairs that each appear in the other's top-k. It then smooths the affinities, raising ROxf Medium mAP from 67.3% to 80.7%. These papers motivate the proposal: they show global similarity statistics are polluted by distant pairs. We reuse their masking and smoothing, but we apply them to a new object. They compare input kernels; we compare task gradient kernels inside a deep multi-task network. Multi-Task Learning Using Neighborhood Kernels builds task similarities over neighborhoods in shallow kernel machines. It is the closest conceptual ancestor. It neither measures conflict inside a shared deep representation nor acts on it.

The multi-task direction supplies both the problem and the strongest competitor. RotoGrad (conference version) balances gradient magnitudes and learns per-task rotations of the shared features. It trains the rotations as a slow "leader" against a fast backbone "follower" and reports a NYUv2 surface-normal median angle error of 20.80° versus 26.16° for plain multi-task training. Its rotation is global per task, which is exactly the assumption we test. It also costs O(Kd³) for K tasks and feature dimension d, which constrains us: we use only a few low-dimensional region rotations. Generative Modeling for Multitask Visual Learning (MGM) couples a multi-task network with one self-attention GAN (a single generator). It supervises synthetic images only through weak scene labels, via a refinement network. We keep MGM's weak-supervision loop but replace its single generator with region-specialized ones. AdaMV-MoE routes inputs to task-dedicated expert blocks. It is the main "just use a mixture of experts" competitor, discussed below.

The multi-generator direction supplies the generators and a warning. MAD-GAN forces K generators apart with a discriminator that must identify which generator made each sample. MEGAN uses a learned gate plus a load-balancing penalty and is sensitive to that penalty's weight, which sometimes leaves generators unused. HC-MGAN transfers a generator-identity classifier to real images to cluster them. All three decide specialization from pixel appearance. We instead assign each generator a region fixed externally by the task-agreement graph, so no load-balancing term is needed. Towards convergence to Nash equilibria in two-team zero-sum games shows that competitive multi-generator training can fail to converge under standard gradient methods. This is a constraint on our design. We therefore avoid a shared multi-class discriminator game and train each regional generator as an ordinary conditional GAN on softly weighted data.

Research Program

The diagnostic. Let $z(x)$ be the shared representation of input $x$, and let $g_k(x) = \partial \mathcal{L}_k / \partial z(x)$ be task $k$'s gradient at that representation. Computing $g_k$ needs one cheap backward pass per task through the heads only. For sample $i$, let $N_i$ be its k-reciprocal neighbors in $z$-space, found in a memory bank of recent samples, and let $G_k[N_i]$ be the Gram matrix of task-$k$ gradients over that neighborhood. The local task alignment between tasks $k$ and $l$ is

$$A_{kl}(i) = \frac{\langle G_k[N_i],\, G_l[N_i]\rangle_F}{\lVert G_k[N_i]\rVert_F\, \lVert G_l[N_i]\rVert_F},$$

where $\langle\cdot,\cdot\rangle_F$ is the elementwise (Frobenius) inner product. We also record the signed mean gradient cosine between $g_k$ and $g_l$ over $N_i$. The first quantity asks whether the two tasks treat nearby samples as alike. The second asks whether they push the shared features the same way. Stacking these values over all task pairs gives an agreement profile $a(i)$ for each sample. We smooth the profiles over the k-reciprocal graph in the style of Cluster-Aware Similarity Diffusion, which prevents isolated noisy samples from forming spurious regions. We then cluster the smoothed profiles into $M$ soft regions. Regions are recomputed every few epochs rather than every step, so they lag the backbone and do not chase it.

Region-conditional rotation. Each task keeps RotoGrad's global rotation. It also gets $M$ low-dimensional rotations of a 64–256-dimensional subspace of the shared features. These are mixed by the sample's soft region weights and trained with RotoGrad's leader–follower schedule. Parameter and compute budgets are matched to RotoGrad by shrinking the global rotation's subspace accordingly.

Region-specialized generators. In the partial-label regime we train $M$ lightweight conditional generators. Each is trained on real images weighted by their membership in its region. Synthetic images inherit their region identity, which selects the region rotation during MGM-style weak supervision. The generators' job is to supply extra training signal exactly in the regions where conflict is worst and labels are fewest. Whether they beat simpler ways of doing this is an explicit test below.

Baselines

The naive reference points are separate single-task networks and plain multi-task training with summed losses. We run both on the same backbone, data splits and epochs. They establish whether negative transfer exists in each split at all. If plain multi-task training already matches the single-task networks on some benchmark, that benchmark cannot distinguish the methods and we report it as such.

The strongest existing method on the conflict question is RotoGrad. We run it with its published leader–follower schedule and the same subspace budget as our region rotations. We give it the same hyperparameter search (equal number of trials over learning-rate ratio and subspace size). The reported number to beat is its NYUv2 surface-normal median angle error of 20.80° from RotoGrad. We rerun it rather than copy the number, because our backbone may differ. Beating it means a lower normal error with no degradation beyond seed noise on depth and segmentation, averaged over at least three seeds.

For the low-label regime the strongest competitor is MGM, run at 50% and 25% labels with its single generator. Its generator gets parameters equal to the sum of our $M$ generators. For MGM on NYUv2 we found no reported result on Lacuna for per-task metrics; the file states only that at 50% labels it matches baselines trained on 100% labels. We therefore reproduce it. Beating it means lower per-task error at 25% and 50% labels at matched generator parameters and GPU hours.

AdaMV-MoE is the input-routed alternative. We give it the same total parameters and training compute (no reported result found on Lacuna for AdaMV-MoE on NYUv2 or CelebA). A MEGAN-style multi-generator variant uses a learned gate and load balancing in place of our agreement-defined regions. It isolates how specialization is decided (MEGAN reports Inception Score 8.33 on CIFAR-10, a generation metric that does not transfer to our setting).

Two ablations isolate the key ingredient. The first defines regions by k-means on the shared features $z$ instead of on agreement profiles, with the same $M$. The second computes agreement globally, with no neighborhood mask.

Why not just…?

Why not just cluster the features with k-means and give each cluster its own rotation, with no generators and no agreement profiles? This is the most serious threat, and our feature-clustering ablation runs exactly this. If feature clusters give the same gains as agreement-defined regions, then the core claim fails: task relationships would not be what defines the useful partition, and we would report that. We expect a gap only if conflict cuts across visual similarity, for example two visually similar corridors where one has reflective floors that break depth. The diagnostic tells us in advance whether that happens. The key number is the mutual information between agreement regions and feature clusters. If it is near its maximum, the two partitions are the same and this alternative wins by default.

Why not just use a mixture-of-experts multi-task network like AdaMV-MoE, which already routes each input to different parameters? Expert routing does make parameters input-dependent. But its gate is trained only on the task losses, with no signal about where tasks conflict. It also cannot add training data to underpopulated conflict regions. The matched-compute AdaMV-MoE comparison at 25% labels settles this. If it matches our method there, the generator component adds nothing and the contribution reduces to the diagnostic.

Why not just oversample the real images in high-conflict regions instead of training generators? In the full-label regime this is probably just as good, and we expect it to be. We therefore include region-reweighted real-data sampling as a baseline in every regime. Generators can only help where real data in a region is too scarce to reweight usefully. MGM's result that synthetic data can stand in for half the labels is the evidence this is plausible. The 25% split is where the test is decisive.

Why not just measure agreement globally, as current gradient-surgery and rotation methods implicitly do? The local-alignment literature shows that global statistics blur distinct regions: 47% versus 96.25% on Digital, and 67.3% versus 80.7% mAP on ROxf. Those results come from different problems, so they motivate the design but do not prove it here. The global-agreement ablation tests it directly.

Proposed Experiments

The first and decisive check is the diagnostic alone, before any method is trained. On NYUv2 (segmentation, depth, normals) we compute the agreement profiles at several training checkpoints of a plain multi-task model. We do the same on CelebA with 40 attributes, where there are many task pairs. We ask what fraction of the variance in pairwise agreement lies between regions rather than within them. We also ask whether some regions show negative normal–depth agreement while the global average is positive. We control neighborhood size by sweeping k, and we use a shuffled-neighborhood null. If between-region variance is not clearly above the null, the program stops there, and the negative result about the homogeneity of task conflict is itself worth reporting.

If the diagnostic passes, we train region-conditional rotation at 100% labels on NYUv2 and CelebA against plain multi-task training, RotoGrad, AdaMV-MoE and the two ablations. We report per-task metrics (normal median angle, depth error, segmentation mean intersection-over-union, CelebA mean F1) over three seeds, plus wall-clock cost.

We then add region-specialized generators at 50% and 25% labels, against MGM, region-reweighted real data and the MEGAN-style gated variant. We extend the best configuration to Tiny-Taskonomy to test scaling to six tasks. Multi-MNIST serves only as a controlled sanity check: there we plant region-dependent conflict by flipping one task's label convention in a known subset of images, and we verify that the diagnostic recovers that subset.

The outcome that would confirm the proposal has three parts. Agreement regions must differ measurably from feature clusters. Region rotations must beat RotoGrad at matched budget on the conflicting tasks. And generators must beat reweighting at 25% labels. Failure on any one localizes where the idea breaks.