Multi-task learning trains several prediction tasks together so that data-rich tasks help data-poor ones. It works only when the tasks sharing parameters are compatible. In suites with dozens or hundreds of tasks, practitioners therefore estimate how much each pair of tasks helps each other, build a graph from those estimates, and partition the graph into groups that each get their own shared network. This proposal argues that every partitioning objective used for this step carries a built-in preference about how large groups should be. That preference, more than the quality of the affinity estimates, often decides whether rare tasks get a group of their own or are absorbed into a dominant family where they suffer.
The imbalanced-clustering literature has shown this preference precisely for ordinary data. Normalized and ratio cuts favor equal-sized parts. Minimum cut favors isolating singletons. k-means splits large dense clusters. The task-grouping literature has not examined it, even though real task suites are visibly lopsided: many dense pixel-prediction tasks next to a few high-level ones, or many similar schools next to a handful of atypical ones. We propose to make the size preference an explicit, tunable variable in task grouping, to measure how it shifts per-task outcomes and especially outcomes for tasks in small latent groups, and to test a specific prediction: the best size preference depends on how noisy the affinity estimates are.
Core Claim
Grouping objectives fall into two kinds by their size preference. Summed within-group affinity favors merging, which is why the mixed-integer program in Towards Principled Task Grouping for Multi-Task Learning needs hard minimum and maximum group sizes. Normalized cuts and k-means favor balance. None of these preferences is chosen with the task distribution in mind.
Our claim has two parts. First, when the latent task groups are skewed in size, a balance-favoring objective splits the dominant family. A merge-favoring objective with a size cap either does the same or folds minority tasks into a large group. Either error shows up mostly on the minority tasks. Second, average multi-task metrics hide the damage, because they are dominated by the many tasks in the large group.
The intermediate setting of the size-penalized cut family from Compassionately Conservative Balanced Cuts for Image Segmentation gives one continuous dial. That paper introduced a parameter $\tau$ ranging from near-unnormalized minimum cut to standard normalized cut, with $\tau=1$ as a middle ground. With this dial, the size preference can be swept rather than inherited.
The second, sharper prediction is about noise. Affinity estimates for a small group rest on few task pairs, so they are noisier. As estimation noise rises, the best setting should move toward larger, more balanced groups. With clean affinities it should move toward allowing small groups. If this interaction holds, no single fixed objective is right, and the size preference should be chosen with the estimator's noise in mind.
Relation to Prior Work
Two strands motivate the proposal. The imbalanced-clustering work shows the bias concretely. Clustering and Community Detection with Imbalanced Clusters builds density-adapted graphs and scores them with a minimum-size-constrained cut. On a subsampled USPS 8-vs-9 split, its error drops from 16.67% to 4.8%. The balanced-cuts paper above shows the intermediate setting beats normalized cuts on image segmentation. We reuse their objectives as instruments. We do not claim new cut algorithms.
On the transfer side, three groups of methods perform the grouping step that we study:
- Affinity search and integer programming. Towards Principled Task Grouping and Task Grouping for Automated Multi-Task Machine Learning via Task Affinity Prediction estimate affinities, then search over partitions. They fix or ignore group sizes and do not report how minority tasks fare.
- Learned group assignment. DMTG: One-Shot Differentiable Multi-Task Grouping learns task-to-branch assignments by gradient descent with no size term. The research file notes this leaves it open to branch collapse.
- Online affinity thresholds. Selective Task Group Updates for Multi-Task Optimization groups tasks by thresholding online affinity. It was evaluated on balanced vision suites with at most 11 tasks.
Clustered Multi-Task Learning: A Convex Formulation learns task clusters through a convex norm. On MHC-I binding, its recovered groups did not match the known allele supertypes. We treat that mismatch as a possible symptom of the size bias and test it directly.
Injecting Imbalance Sensitivity for Multi-Task Learning addresses a different imbalance: gradient magnitude across tasks within one shared network. It does not address group cardinality. We use it as a competing explanation for why minority tasks suffer, not as a component.
Research Program
Affinity graph. For each benchmark we first build a symmetric task-affinity graph using a cheap estimator that already exists: lookahead transfer gain, in the style of the integer-programming grouping paper. We record the estimator's noise level by repeating estimation across seeds and data subsamples. This yields an empirical noise scale per edge.
Partitioning families. We then partition the same graph with each of the following:
- the balanced-cut family across a sweep of $\tau$;
- the integer program with and without its size bounds;
- average-linkage agglomerative clustering;
- k-means on task embeddings;
- a modularity-based community detector;
- DMTG's learned assignment.
Every resulting grouping is trained with an identical backbone and an identical budget. Each task is scored by its gain over its own single-task model.
Outcome measures. We stratify gains by the size of each task's latent group, using ground truth where it exists and otherwise a consensus partition. Alongside the mean, we report the gain on the worst-off group.
Planted-skew suite. The causal question needs ground truth, so we build a planted-skew suite in two ways:
- synthetic linear tasks whose parameter vectors are drawn around group centers with controlled size ratios, from 1:1 to 20:1;
- reorganized real suites, such as School subsets, where known group labels let us subsample tasks to impose skew.
On this suite we vary two things independently: the size ratio, and the affinity noise, by limiting the data used for estimation. This gives a two-way map from (skew, noise) to the best $\tau$ and to minority-task gain. The map is the main analytic product.
Selection rule. Finally, we propose and test a rule that chooses the size preference by held-out lookahead gain computed during affinity estimation. Choosing the setting this way costs no extra full training runs. Success means matching the best fixed setting chosen with hindsight.
Baselines
Naive reference points. The first is one shared network over all tasks with uniform loss weights. The second is independent single-task training. All gains are measured against single-task training, so it is the zero line. The research file reports, on School, MSE 86.49 for single-task learning. On Taskonomy-11, Selective Task Group Updates reports $\Delta_m$ of −8.65 for plain joint SGD. We will reproduce both under our own budgets rather than rely on the reported values.
Strongest grouping baseline on large tabular suites. This is the predict-then-search method of Task Grouping via Task Affinity Prediction. It reports School MSE 78.7, against 98.43 for its best clustering baseline. On Parkinson and Chemical, no reported result was found on Lacuna for per-dataset metrics. We run its released procedure with the same affinity budget we give our sweep, the same base learners, and the same number of final training runs. Beating it means one of two things. Either the mean MSE matches it while minority-group gain improves with non-overlapping seed intervals, or the mean MSE improves outright.
Strongest grouping baseline on vision. This is Selective Task Group Updates, which reports $\Delta_m$ of −1.42 on Taskonomy-11. On Taskonomy-11 the number of tasks is small enough that we can enumerate groupings. We use this setting as a sanity check that a well-chosen size preference does not hurt when the suite is near-balanced.
Learned and integer-program baselines. DMTG is run with its published temperature schedule. CelebA-40 has no reported quantitative result on Lacuna, so we report our reproduction. The integer program is run with and without its size bounds. This separates the effect of the objective from the effect of the bounds.
Key ablation. The ablation that isolates our ingredient holds the affinity graph fixed and changes only the size preference. If gains appear only when the affinity estimator also changes, our claim fails.
Why not just…?
Why not just use agglomerative clustering, which has no explicit balance term? This is the most serious threat. Average linkage has a size preference too, set implicitly by the merge threshold, and it could match the best $\tau$ across our map. If it does, we will report that the useful lesson is "control the size preference" rather than "use balanced cuts." In that case we will present the noise-dependent selection rule as applying to the linkage threshold equally. The planted-skew map tests both families on identical graphs, so this outcome would be visible rather than argued away.
Why not just tune the number of groups? Changing the group count shifts how many groups there are, not how sizes are distributed among them. A balance-favoring cut asked for four groups on a 20:1 suite will still try to equalize them. We include the group count as a swept variable for every family and report whether size skew in the recovered groups follows the group count or the objective.
Why not just fix the imbalance in optimization with IMGrad? IMGrad reports $\Delta m$ of −4.57% on NYUv2 versus 0.29% for CAGrad, so magnitude balancing clearly matters. It operates inside one shared network, however, and cannot separate a minority family from a dominant one whose gradients are mutually reinforcing. We run IMGrad on the ungrouped suite and as the optimizer inside each grouping. If IMGrad alone closes the minority-task gap on the planted suite, grouping size is not the bottleneck, and we will say so.
Decisive Checks
Main test. On the planted-skew suite at a 10:1 ratio with low affinity noise, normalized-cut grouping should split the large group. Its minority tasks should land at or below single-task performance, while the intermediate setting recovers the planted groups and lifts minority gain. If the minority tasks are fine under normalized cuts, the premise is wrong.
Noise interaction. The best $\tau$ must move toward balance as noise rises. A flat optimum would refute the second claim and reduce the proposal to a fixed recommendation.
Natural test on MHC-I. We check whether recovered allele groups match the supertypes better at the intermediate setting than with the convex cluster norm. That paper reports 8.71% test error, against 11.62% for independent learning. We report both agreement with the supertypes and classification error.
Hidden damage in averages. On School, Parkinson, Chemical, and CelebA-40, the stratified analysis must show that mean metrics hide differences between objectives that appear in the minority strata. If no such hidden differences exist on real suites, the problem is confined to synthetic settings, and the paper becomes a well-bounded negative result.
References
- Towards Principled Task Grouping for Multi-Task Learning
- Compassionately Conservative Balanced Cuts for Image Segmentation
- Clustering and Community Detection With Imbalanced Clusters
- Task Grouping for Automated Multi-Task Machine Learning via Task Affinity Prediction
- DMTG: One-Shot Differentiable Multi-Task Grouping
- Selective Task Group Updates for Multi-Task Optimization
- Clustered Multi-Task Learning: A Convex Formulation
- Injecting Imbalance Sensitivity for Multi-Task Learning