Multi-task learning trains one shared network on several prediction problems at once. It saves compute, but it can also make some tasks worse, a failure called negative transfer. A line of work on task grouping splits the tasks into subsets, each with its own shared network, so that tasks which help each other train together and tasks which hurt each other are kept apart. These methods report large gains. Almost all of them report those gains on a handful of hand-picked task sets: 5 Taskonomy tasks, 9 of the 40 CelebA face attributes, 7 ETTm1 forecasting series. The publication-bias literature has shown how much a method's apparent lead can depend on which datasets are reported and on how carefully the competitors are tuned. This proposal applies that lens to task grouping.
The core idea is simple to state. Instead of evaluating grouping methods on one canonical task subset, we draw many random task subsets from the full task library of each benchmark. On each subset we compare grouping methods against baselines that get the same tuning budget, and we report the whole distribution of gains together with the position of the canonical subset inside it. If the canonical subsets sit in the favorable tail, then the field's picture of how much grouping helps, and of which grouping method is best, reflects a selection effect rather than a general property. We will also publish every trained grouping, including the failures, as a reusable record of negative transfer.
Core Claim
We hypothesize that the reported advantage of task-grouping methods over a well-tuned single shared network, and the reported ranking among grouping methods, are partly artifacts of two things: which tasks were chosen for the benchmark, and how much tuning the baselines received. Both effects can be measured, not just argued. The quantity we estimate for a method $g$ on a benchmark with task library $\mathcal{T}$ is
$$G(g) = \mathbb{E}_{S \sim \mathrm{Unif}\binom{\mathcal{T}}{k}}\left[\, L_{\mathrm{best}}(S) - L_g(S) \,\right].$$
Here $S$ is a random subset of $k$ tasks, $L_g(S)$ is the mean held-out normalized loss over the tasks in $S$ after training the groups that $g$ selects, and $L_{\mathrm{best}}(S)$ is the lower of two losses: one all-in-one network and one set of independent single-task networks, each tuned with the same budget as $g$. The canonical subset $S^*$ gets a percentile rank within this distribution. That rank is our measure of how favorable the benchmark choice was.
Relation to Prior Work
The grouping methods supply both the objects under audit and the machinery we reuse. Which Tasks Should Be Learned Together in Multi-task Learning? established that training all tasks together causes negative transfer. It approximated group quality from early training and from averages over task pairs, and it searched partitions with branch-and-bound on 5 Taskonomy tasks. Efficiently Identifying Task Groupings for Multi-Task Learning, known as TAG, measures from a single joint training run how one task's gradient step changes another task's loss. It reports that grouping beats gradient-balancing tricks. Towards Principled Task Grouping for Multi-Task Learning casts grouping as a mixed-integer program under group-size limits, using a transfer-gain measure that needs no convexity assumptions. Ensemble Prediction of Task Affinity for Efficient Multi-Task Learning, known as ETAP, corrects gradient-based affinities with a regression model fit to a small number of trained groups. Efficient Task Grouping Through Sample-Wise Optimisation Landscape Analysis, known as SCA, estimates affinity without full training. We do not propose a new grouping algorithm. We ask whether the evidence for all of these methods holds up once the task set is not chosen by the authors, and all of them were evaluated on at most about 11 tasks.
Three papers motivate the audit. Over-optimistic evaluation and reporting of novel cluster algorithms showed that searching over datasets and tuning competitors asymmetrically can manufacture a lead. In their Two Moons example, a default-tuned DBSCAN scored 0.0007 AMI against Rock's 0.7881, and a properly tuned DBSCAN scored 0.8300. They also documented that a failure dataset was left out of the paper. Position: Embracing Negative Results in Machine Learning argues that suppressing failures wastes compute and blocks pruning of search spaces, and it sets quality standards for negative results that our protocol adopts. Dead Science Walking shows that corpus bias compounds in automated pipelines. We cite it only as motivation for releasing failure records in a form machines can read.
Two findings constrain the design. The first is that task affinity shifts with architecture and optimization hyperparameters (Standley et al.; Fifty et al.), so the audit must fix and report the backbone and tuning protocol. The second is Clustered Ullmann-style reversal, which appeared in clustering rather than multi-task learning, so whether it transfers to this setting is an open empirical question, not an assumption.
Research Program
We build a sampling-based evaluation harness on benchmarks whose full task library is larger than the canonical subset. CelebA offers all 40 attributes, where the 9-task subset is canonical. Taskonomy offers its wider task list, where the 5-task and 11-task sets are canonical; we will use a reduced-resolution split to make it affordable. QM9 offers 11 properties, and we sample subsets of size 5 to 7. The 10-task Chemical (MHC-I) and Ridership benchmarks allow subsets of size 5. ETTm1 has only 7 tasks, so it serves as a reproduction check rather than a sampling target.
On each benchmark we draw 30 to 50 subsets at the canonical size $k$, and we always include the canonical subset. On each subset we run every grouping method's affinity estimation and selection step. We then train the chosen groups to convergence under a fixed backbone. Every method and baseline gets the same hyperparameter search: a Tree-structured Parzen Estimator (a standard Bayesian hyperparameter search) with an equal number of trials, matched in GPU-hours. The search budget covers affinity estimation, so a method that is cheap to estimate but expensive to train is charged fairly.
The analysis has three parts. The first is the distribution of $G(g)$ and the percentile of $S^*$. The second is how stable method rankings are across subsets, measured with Kendall's tau between per-subset rankings. The third is a decomposition of each method's canonical-subset lead into a task-selection part and a tuning-asymmetry part. To obtain that decomposition, we rerun the canonical subset with the original papers' baseline settings and compare it to the equal-tuning run.
Every trained group becomes one entry in a released ledger of task subset, grouping, per-task loss, and configuration. The ledger records negative transfer as data, not as something to leave out of the paper. As one secondary test of its value, we refit ETAP's residual regression twice: once on its usual exploration samples, and once with ledger entries showing severe negative transfer added. We then measure whether its correlation with true gain holds up on subsets it has never seen.
Baselines
The naive reference point is a single shared network trained on all $k$ tasks with standard summed losses, together with independent single-task networks. Each receives the full equal tuning budget, and $L_{\mathrm{best}}$ takes whichever is better on each subset. A grouping method counts as beating it on a benchmark if the 95% bootstrap interval of $G(g)$ over subsets excludes zero, not merely if it wins on $S^*$.
The strongest existing grouping methods are TAG, the mixed-integer program of Wang et al., and ETAP. All are run from released or reimplemented code, using the affinity sampling intervals the authors recommend. The best number reported for TAG is a 10.0% test-loss reduction relative to training all tasks together on Taskonomy (Fifty et al.). ETAP reports a Pearson correlation of 0.84 with true gain on ETTm1, against 0.47 for TAG (ETAP). SCA reports relative improvements of 45% and 23.1% over standard multi-task learning and Nash-MTL on CelebA (SCA). For the Wang et al. mixed-integer program, no reported loss number was found on Lacuna. Reproducing these figures on the canonical subsets is a precondition for the audit. Our finding is the gap between these figures and the subset-averaged, equal-tuning values.
The strongest non-grouping competitor is Selective Task Group Updates (Jeong et al.). On 11-task Taskonomy it reports a $\Delta_m$ of −1.42, against −8.65 for SGD, −6.22 for Aligned-MTL, and −7.58 for FAMO. $\Delta_m$ is the average per-task change relative to single-task models, so values closer to zero are better. We run it and PCGrad inside a single shared network under the same budget.
The key ablation isolates each ingredient. Holding the canonical subset fixed while equalizing tuning isolates tuning asymmetry. Holding tuning fixed while resampling subsets isolates task selection. A random-grouping control with the same group sizes tells us whether a method beats chance on uncurated subsets.
Why not just…?
Why not just run the existing methods once on all 40 CelebA attributes? That is a useful single data point, and we will include it. But it answers a different question: how the methods behave at a larger scale, where solvers and capacity limits change the picture. It says nothing about how favorable the canonical 9-task choice was at the size the literature actually reports. Only sampling at fixed $k$ separates task selection from scale.
Why not just tune the baselines harder, as Ullmann et al. did? That handles one of the two mechanisms, and it is a real threat to the sampling arm's importance. If equal tuning alone removes most of each method's lead on $S^*$, then subset selection matters less than we hypothesize. The decomposition experiment reports exactly this split. We will state that result plainly if it is what we find.
Why not skip grouping and use gradient surgery or loss reweighting? Fifty et al. found that grouping beats GradNorm and PCGrad, and Jeong et al. report that MGDA and Nash-MTL converge poorly at scale. Both findings come from curated subsets, however, and whether they hold on random subsets is one of the rankings we test. Including Selective Task Group Updates and PCGrad as competitors settles the question directly.
Why not accept that canonical subsets were chosen for compute, not advantage? The authors' intent is irrelevant to the measurement. A subset chosen for innocent reasons can still be unrepresentative. The percentile of $S^*$ is informative either way, and a median percentile would be a publishable negative result in favor of the current benchmarks.
Decisive Checks
The main hypothesis is supported if, on at least two of CelebA, Taskonomy and QM9, the canonical subset ranks above the 75th percentile of grouping gain. It is also supported if the subset-averaged, equal-tuning gain of the best grouping method is less than half its canonical-subset gain, or if between-method Kendall's tau across subsets falls below 0.3.
The hypothesis is refuted if canonical subsets fall near the median and rankings stay stable. In that case the ledger and the harness still ship, as evidence that task-grouping results generalize.
The ledger's value as a record of negative transfer is supported if refitting ETAP with the negative-heavy entries raises its correlation with true gain on unseen subsets by a margin outside the variation across random seeds. If it does not, we report that too.
References
- Which Tasks Should Be Learned Together in Multi-task Learning?
- Efficiently Identifying Task Groupings for Multi-Task Learning
- Towards Principled Task Grouping for Multi-Task Learning
- Ensemble Prediction of Task Affinity for Efficient Multi-Task Learning
- Efficient Task Grouping Through Sample-Wise Optimisation Landscape Analysis
- Over-optimistic evaluation and reporting of novel cluster algorithms: an illustrative study
- Position: Embracing Negative Results in Machine Learning
- Dead Science Walking: Publication Bias and the AI Scientist Pipeline
- Selective Task Group Updates for Multi-Task Optimization