Frequency-Band Gradient Arbitration for Mixed Local–Global Multi-Task Learning

When one network learns several tasks at once, each task pushes the shared features in its own direction, and the tasks often disagree. Existing fixes treat these pushes as flat vectors over all parameters. They detect disagreement by measuring the angle between task gradients, and they resolve it by choosing one compromise direction. That bookkeeping ignores something visual tasks obviously differ on: spatial scale. Segmentation and edge detection depend on sharp, high-frequency detail. Depth layout and whole-image recognition depend on coarse, low-frequency structure. Two tasks may agree on the coarse layout of a scene yet fight over fine texture. A single cosine computed across all scales averages that agreement and that fight together, and the compromise it produces gives up on both.

This note proposes to make the compromise per spatial frequency band. For each task we take the gradient of its loss with respect to the shared feature map, split that gradient into a few frequency bands with a fixed Fourier mask, run a conflict-averse aggregation separately within each band, and backpropagate the recombined gradient into the encoder. Two things are new here. The first is a measurement: where across scales multi-task conflict actually lives. The second is an optimizer that uses that structure at almost no extra cost, because per-task gradients at a feature map only require backpropagating through the small task heads.

Core Claim

Our claim is that in multi-task dense prediction, and especially when dense local tasks are mixed with global image-level tasks, gradient conflict is concentrated in specific spatial frequency bands rather than spread uniformly. If that is true, resolving conflict band by band preserves the agreeing coarse-scale signal that whole-vector methods throw away. That should beat whole-vector conflict-averse descent at equal compute.

Here is the mechanism stated precisely. Let $g_t$ be the gradient of task $t$'s loss with respect to the last shared feature map. Let $P_1,\dots,P_B$ be orthogonal projectors onto $B$ radial frequency bands of that map, implemented as FFT masks that sum to the identity. In each band we solve the conflict-averse subproblem of Conflict-Averse Gradient Descent for Multi-task learning on the band-restricted gradients:

$$d_b = \arg\max_{d}\ \min_t \langle P_b g_t, d\rangle \quad \text{s.t.}\quad \|d - \bar g_b\| \le c\,\|\bar g_b\|,$$

where $\bar g_b$ is the average of the $P_b g_t$ over tasks and $c$ is CAGrad's radius hyperparameter. The update sent into the encoder is $d=\sum_b d_b$.

The bands are orthogonal and the per-band averages sum to the full average gradient. As a result, the total deviation from the average gradient is still bounded by $c\,\|\bar g\|$. This is the quantity CAGrad's argument for converging to the average-loss minimum relies on, though restating that proof for feature-space updates is part of the work. The same bound shows what the method gains: each band gets its own worst-case protection, so a task that conflicts only at high frequency can no longer veto low-frequency progress.

Research Program

The first stage is purely diagnostic, and it decides whether the rest is worth doing. We train standard multi-task models, record per-task feature gradients at the shared layer, and track cosine similarity between task pairs within each band over the course of training. The models are an HRNet convolutional backbone and the InvPT transformer backbone used in Multi-task Learning with 3D-Aware Regularization, plus a hybrid convolution–attention encoder in the style of The Fully Convolutional Transformer for Medical Image Segmentation. We predict three patterns. Segmentation and depth should align in the lowest band and diverge in the highest. Normals and edges should conflict with segmentation mainly at mid-to-high frequencies. A global task should agree with dense tasks only near the DC component. That global task is multi-label class presence, derived directly from the segmentation masks.

If conflict turns out to be uniform across bands, we report that as a negative result about where interference lives, and we do not proceed to the optimizer.

The second stage builds the band-wise optimizer described above. It applies the band split at one or two shared layers and varies $B$ between 2 and 4. We compare fixed log-spaced band edges against edges set from each layer's feature power spectrum.

The third stage targets the heterogeneous setting that Joint-Task Regularization for Partially Labeled Multi-Task Learning explicitly leaves open, where pixel-level and image-level outputs are trained together. That paper (JTR) regularizes stacked dense predictions through a shared autoencoder, which cannot take a scalar global head. Our method needs no such stacking, because it operates on gradients at the shared features.

The other cited works play distinct roles. CAGrad supplies the per-band solver. Improving Multi-Task Generalization via Regularizing Spurious Correlation motivates the work: it shows that tasks borrow shortcut features from one another through shared representations. It handles this with identical experts and environment-based invariance; we instead ask whether the borrowing happens at a particular scale. The 3D-aware regularizer is complementary rather than competing. It adds a geometric training signal, and our method changes only how conflicting signals are combined, so the two can be stacked.

Baselines

The naive reference point is equal-weight linear scalarization of task losses, with the weights grid-searched on validation data. Every method gets the same total budget of tuning runs, and all runs use the same backbone, schedule, augmentation, and data splits.

The strongest existing optimizer is CAGrad. On NYU-v2 it reports a per-task performance drop $\Delta m\%$ of 0.20%, against 3.97% for PCGrad and 5.59% for MTAN (Conflict-Averse Gradient Descent for Multi-task learning). We run it two ways. One is the original parameter-space version. The other is a feature-space version that aggregates at the same layer as our method, with no band split. That second version gives identical cost and isolates the band split as the only difference. Beating CAGrad means a lower $\Delta m\%$ than feature-space CAGrad across three seeds, with non-overlapping standard errors, at matched wall-clock time and $c$ tuned separately for each method.

The strongest architectural regularizer on these benchmarks is the 3D-aware regularizer. On InvPT for NYU-v2 it reports segmentation improving from 53.56 to 54.87 mIoU and depth RMSE falling from 0.5183 to 0.5006, and on PASCAL-Context it reports +1.51 mIoU on part segmentation (Multi-task Learning with 3D-Aware Regularization). We do not claim to beat it. We test whether combining it with band-wise arbitration adds further gain over the regularizer alone.

For Cityscapes, the file reports no per-task numbers for CAGrad or JTR, so we run both ourselves: no reported result found on Lacuna for CAGrad or JTR per-task scores on Cityscapes.

Two ablations isolate the key ingredient. The first, random-subspace arbitration, splits the feature gradient into $B$ random orthogonal subspaces of the same sizes as the frequency bands. It tests whether frequency structure matters or whether any extra degrees of freedom in the compromise would help. The second, operator-split arbitration, runs CAGrad separately on the convolutional and the attention parameters of the hybrid encoder.

Why not just…?

Why not just tune the loss weights? Fixed weights scale each task's whole gradient. They cannot let a task win at low frequency while yielding at high frequency. CAGrad's NYU-v2 gap over fixed-weight MTAN (0.20% versus 5.59% $\Delta m$) already shows that weighting alone leaves performance unused. The grid-searched scalarization baseline measures how much of our gain, if any, weighting recovers.

Why not just run CAGrad separately on the local (convolution) and global (attention) branches? This is the most serious threat. The tension highlighted by Scalable Multi-Task Low-Rank Model Adaptation supports it: that paper (mtLoRA) reduced inter-task conflict from 60.6% to 53.6% purely by moving adaptation away from attention projections, which suggests architecture-aligned splits may capture most of the structure. Our counter-hypothesis is that operators do not map cleanly onto scales. Convolutions with large receptive fields carry low-frequency content, and attention carries edges. The operator-split ablation settles the question directly. If it matches band-wise arbitration, the contribution shrinks to the diagnostic, and we will say so.

Why not just give each task more private capacity, such as task-specific decoder branches? Extra private capacity reduces conflict in the shared layers by sharing less. It also gives up the regularizing benefit of sharing that motivates multi-task learning in the first place, and it costs parameters at inference, which our method does not. We include a capacity-matched variant with private branches. Our method should beat it on held-out generalization, measured on NYU-v2 test and on Cityscapes, not only on training-distribution scores.

Proposed Experiments

The main benchmarks are NYU-v2 (segmentation, depth, normals), Cityscapes (segmentation, disparity), and PASCAL-Context (semantic and part segmentation, edges, normals). Each is evaluated in two settings: dense tasks only, and dense tasks plus a mask-derived global class-presence task. Metrics are per-task mIoU, depth RMSE, normal angular error, edge F-score, and mean average precision for the global task, summarized by $\Delta m\%$ relative to single-task models. We also report training wall-clock time and memory.

There are three decisive checks. First, the stage-one diagnostic must show band-concentrated conflict. Second, band-wise arbitration must beat both feature-space CAGrad and random-subspace arbitration, which tests whether frequency structure specifically drives the gain. Third, the advantage must be larger in the mixed local–global setting than in the dense-only setting, because that is where the hypothesis predicts the strongest scale-specific disagreement.

As an exploratory medical check, we would repeat the diagnostic on Synapse multi-organ CT. Small organs and large organs plausibly conflict at different scales there. PVT-GDLA reports an 85.32 average Dice on Synapse (Gated Differential Linear Attention: A Linear-Time Decoder for High-Fidelity Medical Segmentation), which serves as a single-task reference.