Where Tasks Collide Inside Attention: Flow-Guided Forking for Multi-Task Transformers

When one network is trained on several tasks at once (multi-task learning), some tasks help each other and some interfere. Current fixes act on the whole network at once. They either rescale each task's loss or gradient, or they rotate the final feature vector just before the task-specific output layers. None of them asks where inside the network the tasks actually fight. For Transformers there is a principled way to ask. Stacked self-attention layers move information like a random walk run backwards over tokens, so we can trace, for each task, which attention edges carry the information that task's output depends on. Our bet is that interference concentrates on a small set of edges: places where two tasks route heavily through the same connection but their gradients push that connection in opposite directions.

We propose to measure this "flow collision" per layer and per head during training. We then use it to decide the one design choice that matters most for cost in a shared Transformer: the depth at which each task gets its own version of the attention pattern. Before that depth, every task shares one forward pass. After it, each task needs its own. Placing that fork late and only where collisions occur should recover most of the benefit of task-specific attention at a fraction of the inference compute. If flow collision predicts this depth better than simple per-layer gradient statistics, it is a useful new tool for multi-task learning. If it does not, the random-walk view adds nothing here, and we will be able to say so cleanly.

Core Claim

The claim has two parts. The first is a diagnostic claim: a flow-weighted collision score, built from attention rollout and task gradients on attention weights, ranks layers by how much a shared model gains from making that layer task-specific. It should rank them more accurately and earlier in training than per-layer gradient cosine similarity or raw attention statistics. The second is an intervention claim. Adding cheap task-specific head-mixing matrices only at and above the flow-chosen fork depth matches, or beats, per-task mixing at every layer, and it beats output-level de-confliction. It should do this while sharing substantially more forward computation.

Grounding in Prior Work

The mechanism rests on three results from the attention-as-graph line. What Dense Graph Do You Need for Self-Attention? proves that information transfer through stacked attention layers is equivalent to a reversed random walk on the attention graph. It uses random-walk capacity to design sparse attention, and treats GLUE only as a post-pretraining evaluation. We use the same equivalence for a different purpose: to define per-task reachability inside a model trained jointly. Quantifying Attention Flow in Transformers supplies the concrete computation. It forms attention rollout on $A+I$ to account for residual connections, and shows that rollout tracks token importance far better than raw attention in deep layers (~0.71 versus ~0.29 in DistilBERT layer 6). That result also constrains us: raw attention is not an adequate signal. How Does Reasoning Flow? shows that flow through an attention graph picks out causally important tokens in a single-task reinforcement learning setting: masking high-flow tokens reverses 14.9% of answers versus 0.5% for low-flow tokens. It uses flow to reweight credit for one objective. We use flow from several task outputs at once to find where their demands clash.

The intervention comes from Coeff-Tuning. It treats each attention head as a graph filter and learns a small unconstrained head-mixing matrix $\alpha \in \mathbb{R}^{H \times H}$, where $H$ is the number of heads. This lets heads be combined, including subtractively, which escapes the softmax's restriction to convex combinations, at about 0.001% added parameters. Coeff-Tuning adapts one task at a time and never trains tasks jointly. We give each task its own $\alpha_k$ inside a jointly trained model, and use flow collision to choose where those matrices go.

On the multi-task side, two papers motivate the work and define the bar. RotoGrad resolves conflicting gradient directions by rotating the final shared feature per task. It treats the backbone as opaque and pays a cost cubic in feature width. Dual-balancing for multi-task learning (DB-MTL) balances loss scales with a log transform and equalizes gradient norms. It is simple and strong, but it does not touch gradient direction. AutoMTL searches over share, branch, or skip choices per operator, and was built and tested only on convolutional networks. Our fork-depth choice is the Transformer analogue of that decision, made by measurement instead of search.

Research Program

We take a shared Transformer backbone with $K$ task heads. Each task $k$ reads out from its own set of output tokens: the class token for classification, all patch tokens for dense prediction.

For each layer $l$ we compute the reversed-walk reach $\rho^{(l)}_k(i)$. This is the probability that a walk starting at task $k$'s readout tokens and moving down through the residual-augmented attention matrices of the layers above $l$ lands on token $i$. It is obtained by partial rollout. The walk mass that task $k$ sends through attention edge $(i \to m)$ at layer $l$ is then $f^{(l)}_{k,im} = \rho^{(l)}_k(i)\,A^{(l)}_{im}$, where $A^{(l)}$ is that layer's attention matrix. Backpropagation already produces $\partial L_k / \partial A^{(l)}_{im}$, the gradient of task $k$'s loss with respect to the edge weight. We define the collision between tasks $j$ and $k$ at layer $l$ as

$$C^{(l)}_{jk} = \sum_{i,m} f^{(l)}_{j,im}\, f^{(l)}_{k,im}\; \mathbb{1}\!\left[\frac{\partial L_j}{\partial A^{(l)}_{im}} \cdot \frac{\partial L_k}{\partial A^{(l)}_{im}} < 0\right].$$

The same sum restricted to one head gives a per-head score. The score is large only when both tasks depend heavily on the same edge and want it changed in opposite directions. The downstream reach term $\rho$ is what separates this score from a local gradient statistic, because it discounts edges whose information never reaches a task's output.

We compute $C$ every few hundred steps on a small batch. It needs the attention maps, their gradients, and one rollout per task, so we expect overhead of the same order as FlowTracer's reported 2.2–4.5%; we will measure it rather than assume it.

The intervention works as follows. We train with hard parameter sharing, with DB-MTL loss balancing turned on by default so that magnitude effects are removed. After a short warm-up we choose the fork depth $l^\star$ as the shallowest layer whose collision exceeds a threshold, calibrated once on a validation split. From $l^\star$ upward, each task gets its own head-mixing matrix $\alpha_k$, and the layers below stay fully shared.

In the analysis we examine whether collisions sit on specific heads and middle layers, which the flow literature suggests. We also check whether they move during training, and whether task pairs known to conflict show higher $C$. On NYUv2, RotoGrad reports that surface normals degrade the other tasks under plain joint training.

Baselines

The naive reference is plain hard sharing with equal loss weights on the same backbone. Next to it we report single-task models, so that negative transfer is visible directly. The strongest existing optimizer-level method is DB-MTL, which the file describes as best across NYUv2, Cityscapes, QM9, and the Office datasets. The file quotes no exact per-task figure for it, so there is no reported result on Lacuna for DB-MTL's NYUv2 numbers to copy, and we rerun it. The strongest direction-level method is RotoGrad. On NYUv2 it reports a median surface-normal angle error of 20.80 against 26.16 for vanilla joint training, but on its own convolutional backbone, so that number sets expectations rather than serving as a direct comparison. We also run PCGrad, which projects out conflicting gradient components.

On the architectural side, the strongest competitor is per-task head mixing at every layer: Coeff-Tuning's $\alpha$ made task-specific throughout. It is the upper bound on what our intervention can reach, and it is also the most expensive in shared compute. The key ablations give the same per-task $\alpha_k$ at a fork depth chosen three other ways: uniformly at random, by per-layer gradient cosine between tasks, and by the collision score with the reach term removed. The last one isolates the random-walk ingredient.

All methods use the same backbone initialization, data, number of training steps, and a matched hyperparameter search budget. Any trial count spent calibrating our threshold is charged to our method.

Beating the baselines means two things. First, a higher overall relative improvement over single-task learning ($\Delta_p$, the average percent change across task metrics) than DB-MTL and RotoGrad. Second, $\Delta_p$ within noise of per-task mixing at every layer, while sharing at least half the layers' forward computation. Across seeds, the flow-chosen fork must also outperform the gradient-cosine-chosen fork.

Why not just…?

Why not just balance losses and gradient norms with DB-MTL? DB-MTL fixes gradient scale, not direction. RotoGrad's NYUv2 surface-normal gap (20.80 versus 26.16) shows that directional conflict carries real error that scaling leaves behind. Still, DB-MTL could absorb most of the gain on a given benchmark. That is why it runs underneath our method as well. If adding flow-guided forks on top of DB-MTL does not raise $\Delta_p$, the intervention claim fails.

Why not just compute gradient cosine similarity per layer and fork where it is most negative? This is the most serious threat, because it needs no rollout at all. The argument for flow is that a local gradient conflict on an edge whose information never reaches a task's output is harmless, and only the reach term knows that. The fork-by-gradient-cosine ablation and the collision-without-reach ablation settle this directly. If either matches the full score, we will report that the random-walk component is unnecessary for this purpose.

Why not just give every task its own head-mixing matrix (or its own low-rank adapter) at every layer? These are cheap in parameters, so parameter count is not our argument. The argument is compute. Once any task's attention differs at layer $l$, every layer above must run once per task. Mixing everywhere means $K$ full forward passes, while a late fork shares everything below $l^\star$. The Pareto comparison of $\Delta_p$ against inference FLOPs shows whether the saved compute costs accuracy. Per-task low-rank adapters face the same compute issue, plus the convex-combination limit that Coeff-Tuning identifies.

Why not just search for the branching point, as AutoMTL does? Search can find a good fork, but it costs many trained supernetworks, and AutoMTL has only been built for convolutional networks. We include an oracle sweep, training one model per possible fork depth. This both judges whether our score picks near-optimal depths and gives an upper bound that a search method could at best reach.

Proposed Experiments

We run three benchmarks. The first is NYUv2 (segmentation mIoU, depth absolute relative error, surface-normal median angle) with a Vision Transformer encoder and light dense decoders. The second is Cityscapes (segmentation and depth), to test transfer of the calibrated threshold. The third is GLUE trained jointly on a pretrained BERT-sized encoder, where the only reported multi-task reference in the file is the original GLUE BiLSTM result of 69.0 multi-task versus 64.8 single-task. That comparison is between architectures and serves as context only. We add CelebA's 40 attribute tasks to test how the pairwise score and the fork decision scale with $K$. There the pairwise score costs $K^2$ inner products over edge flows, and we will report wall-clock cost.

The first decisive check is predictive. From the oracle sweep, each layer has a true gain from forking there. We compute the rank correlation between that gain and each score measured at 10% of training: full collision, collision without reach, gradient cosine, and raw attention divergence. The diagnostic claim holds only if full collision wins, with confidence intervals over five seeds, on at least NYUv2 and GLUE.

The second check is causal. At a matched fork depth and matched parameter count, we compare giving task-specific $\alpha_k$ to the heads with the highest collision against the heads with the lowest. A sizable $\Delta_p$ gap supports localization; no gap means interference is diffuse and the premise is wrong.

The third check is the accuracy–compute trade-off against DB-MTL, RotoGrad, PCGrad, and mixing at every layer. It is reported as $\Delta_p$ versus measured inference FLOPs and training overhead. A negative result on the first check would still be informative, since it would bound what random-walk views of attention can say about how tasks share a Transformer.