Narrow Per-Task Readouts as a Defense Against Cross-Task Shortcuts

When one network is trained to do several jobs at once, the jobs share an internal representation. Doing so usually helps. It can also hurt in a specific way: a feature that one task genuinely needs becomes strongly encoded, and a second task starts relying on it because it happens to correlate with that task's labels in the training data. At test time the correlation breaks and the second task fails. Existing defenses against this "shortcut borrowing" are heavy. They use mixtures of experts, decorrelation penalties, and invariance penalties that require the data to be split into labelled environments. This note asks whether a much cheaper structural constraint does the same job. The constraint is to force each task to read the shared representation only through its own narrow, low-rank window.

The idea comes from bottleneck feature extraction, the older line of work that deliberately squeezes a network through a narrow layer. In speech, such layers discard channel noise and reverberation and keep the dominant structure of the signal. Our hypothesis is that a per-task bottleneck behaves like a per-task filter. A task with a small budget of directions must spend them on the directions that most strongly predict its own labels. Peripheral directions amplified by other tasks are left out. We test this in a setting where the shortcut is planted and measurable. We then check whether the effect holds on standard dense-prediction benchmarks, and we measure what it costs, because bottlenecks are also known to destroy information about unusual inputs.

Core Claim

Let $z \in \mathbb{R}^d$ be the shared feature produced by the backbone. Each task $k$ sees only $u_k = P_k z$, where $P_k$ is a learned $r_k \times d$ projection with $r_k \ll d$. The task head $h_k$ acts on $u_k$ alone. The claim is that choosing $r_k$ near the intrinsic dimensionality of task $k$'s own signal does two things. It reduces how much task $k$'s predictions depend on features that other tasks have amplified in $z$. In doing so it recovers much of the out-of-distribution robustness that Improving Multi-Task Generalization via Regularizing Spurious Correlation obtains, without that method's environment labels or second-order invariance gradients.

We state this sharply enough to be falsified. A hard rank limit should beat equally regularized wide heads at matched in-distribution accuracy. If a soft penalty with the same effective rank does equally well, the conclusion becomes "low rank matters, the hard structure does not." That is still a useful answer, but a different one.

This differs from the closest existing work as follows:

  • VMT-Adapter routes all tasks through one shared down/up-projection bottleneck plus elementwise task modulation. Its bottleneck is shared and statically sized, and it is motivated by parameter efficiency rather than shortcut suppression. Ours is per-task and sized per task. - Multi-Task Adversarial Network Bottleneck Features uses a bottleneck to remove a known nuisance (noise type) through an adversarial game. We remove unknown cross-task nuisances with no adversary.

  • TADIB prunes whole modality-to-task links with an information bottleneck. We operate inside a single-input backbone at the level of feature directions.

Research Program

Architecture

The model is a standard shared encoder, followed by one learned rank-$r_k$ projection per task, followed by unchanged task decoders. The cost is $r_k \cdot d$ parameters per task, which is smaller than a typical head.

Choosing the widths

We choose each $r_k$ in two ways and compare them:

  • Validation sweep. A per-task sweep over $r_k$ on held-out data.
  • Single-task heuristic. Set $r_k$ to the stable rank of task $k$'s head-input feature covariance, measured in a short single-task run.

Dimensionality-induced information loss of outliers uses stable rank to locate where networks collapse dimensionality. Here we reuse it as a sizing rule. If the heuristic lands near the validation optimum, the method needs no extra tuning.

Measuring shortcut borrowing directly

Borrowing is usually only inferred from test accuracy. We measure it in two ways:

  • Decodability. Train a linear probe on $u_k$ to predict another task $j$'s target, or a planted cue. Compare against the same probe trained on the unrestricted $z$.
  • Intervention. Ablate the top directions of $z$ that are most predictive for task $j$, and record the change in task $k$'s loss.

Both measures work without environment labels. If borrowing really drives negative transfer, these numbers should track the out-of-distribution gap.

The cost side

The outlier paper shows that low-rank layers project unusual inputs out of the learned subspace, leaving only the training set's bias. That paper constrains our design: it predicts that narrow readouts may make each task more overconfident on inputs that are unusual for that task. We therefore track per-task out-of-distribution detection alongside accuracy.

Secondary payoff

Because each $u_k$ is low-dimensional, task-specific rotations in the style of RotoGrad become cheap. RotoGrad's authors had to restrict rotation to 1,024 dimensions. With narrow readouts, the rotation can act on the full $r_k$-dimensional readout. We test this pairing only once the main claim holds.

Baselines

The naive reference point is a shared encoder with full-width task heads trained on the summed loss. The single-task models trained separately are also reported. The standard multi-task gain $\Delta_m$, the average per-task improvement over single-task models, is computed against them. As a published anchor, DeMT reports $\Delta_m$ of 3.36% and 46.36 segmentation IoU on NYUv2 with a Swin-T backbone, versus 0.31% for MQTransformer. We will reproduce our own shared-backbone numbers rather than compare against these across codebases.

The strongest competitor on this specific question is MT-CRL from Improving Multi-Task Generalization via Regularizing Spurious Correlation. It combines a multi-gate mixture-of-experts backbone with output decorrelation and graph-invariant risk minimization. It targets exactly the cross-task spurious correlation we target, and it reports a 5.5% average generalization improvement across five benchmarks including NYUv2. We run it with its environment partitions on the same backbone, data, and epochs. Our method gets the same hyperparameter-search budget in GPU-hours, with the $r_k$ sweep counted against that budget. Beating MT-CRL means matching its out-of-distribution accuracy within noise across three seeds while using no environment labels. That would be a meaningful result because our method is much simpler. Clearly exceeding it on the planted-shortcut test would be a stronger result.

We also run three further comparisons:

  • Shared bottleneck. A VMT-Adapter-style single bottleneck of total width $\sum_k r_k$, which isolates per-task from shared bottlenecks.
  • Gradient methods. PCGrad, and RotoGrad on unrestricted features. The file reports no number on Lacuna for either on our planted-shortcut setup.

Ablations

The ablation that isolates the key ingredient keeps full-width heads and adds a nuclear-norm penalty on the first head layer. The penalty is tuned so that the head's effective rank matches $r_k$. A second ablation uses parameter-matched wide heads with heavy weight decay and dropout.

Why not just…?

Why not just tune weight decay and dropout on the task heads? This is the most serious threat. Narrow readouts are a form of capacity control, and generic regularization might suppress weak borrowed features equally well. The parameter-matched, heavily regularized wide-head ablation answers this directly. If it reaches the same out-of-distribution accuracy on the planted-shortcut test, we will report that the effect is ordinary regularization and drop the structural claim. Our reason to expect a difference is that weight decay shrinks all directions together, whereas a rank limit forces a choice among them. The decodability probe distinguishes the two mechanisms: under weight decay the planted cue should stay linearly decodable from the head input, and under a rank limit it should not.

Why not just apply PCA or a whitening transform to the shared features? A projection that ignores the task keeps the highest-variance directions, and a feature amplified by another task is likely to be exactly such a direction. Within the bottleneck literature, Subspace models for bottleneck features found that a linear Karhunen-Loève transform gave only about 6.3% average relative gain, versus 31.9% for model-based subspace modeling. That result is on speech recognition, not on shortcuts. We include a fixed per-task PCA readout of the same rank as a baseline. We expect it to fail on the planted shortcut because the cue will carry high variance.

Why not just use one shared narrow bottleneck, as VMT-Adapter does? A shared bottleneck keeps whatever is jointly useful, and a feature one task genuinely needs is jointly useful by construction. A shared bottleneck therefore preserves the shortcut for every task. VMT-Adapter's +3.96% relative gain over full fine-tuning on PASCAL-Context shows that shared bottlenecks are efficient. It says nothing about shortcuts. The matched-width shared bottleneck baseline tests whether separating the windows per task is what matters.

Why not just use gradient surgery? MT-CRL's authors report that PCGrad and GradVac underperform representation-level fixes on their benchmarks. Gradient conflict is also not the mechanism here: a borrowed shortcut can have gradients that agree across tasks during training. We include PCGrad so this argument is tested rather than assumed.

Proposed Experiments

Planted shortcut on NYUv2

The decisive experiment uses a controlled shortcut. In NYUv2 training images we render a faint, uniform color tint whose hue encodes the image's mean-depth bin. The tint is present at both training and test time and is legitimately predictive for depth. In training, it is additionally aligned with the presence of a chosen set of segmentation classes. The test set breaks that alignment.

The depth objective pushes the encoder to encode the tint strongly, so segmentation is tempted to borrow it. The measurements are:

  • segmentation IoU on the broken-alignment test set;
  • the gap between the multi-task and single-task models;
  • tint decodability from each segmentation readout.

The prediction is concrete. Shared full-width heads should show a larger out-of-distribution gap than single-task segmentation. Narrow per-task readouts should close most of that gap, at in-distribution IoU within one point of the wide model. The nuclear-norm and weight-decay ablations decide whether hard rank is what produces this.

Natural benchmarks

We next run NYUv2 (segmentation, depth, normals) and PASCAL-Context (semantic and part segmentation, normals, saliency) without planted cues, reporting $\Delta_m$ and per-task metrics. The questions are:

  • Does narrowing hurt clean performance?
  • Does cross-task decodability fall?
  • Does the stable-rank heuristic land near the validation-optimal $r_k$?

Out-of-distribution cost

Following the outlier paper's protocol, which reports 0.936 AUROC for a feature-based detector and 0.978 for a dimensionality-aware projection with VGG-13 on CIFAR-10 versus MNIST, we measure per-task detection AUROC for both the wide and the narrow models. We also measure the stable rank and alignment of representations at each readout. A clean robustness gain paired with a large AUROC drop would show that narrow readouts trade one failure for another. We would report that tradeoff as the main finding instead of hiding it.

Optional extension

Only if the main result holds, we compare RotoGrad applied inside the narrow readouts against RotoGrad's own truncated-subspace version, on accuracy and wall-clock cost.