Growth-Matched Multi-Task Learning on Unbounded Inputs

When one network is trained to predict several quantities at once, the tasks often behave very differently as inputs become extreme. A molecule's total energy grows with its size while its dipole moment does not. A household's peak electricity draw grows with the scale of its consumption curve while a yes/no "exceeds threshold" label stays bounded. Theory for approximation on unbounded domains says each such target lives in its own function space, defined by how fast it may grow at infinity, and that a network can only approximate it everywhere if its activation grows at a matching rate (Hatano et al., 2024). Multi-task learning ignores this. It shares one feature extractor across all tasks and balances them with a single scalar weight per task. We argue that a large share of the "negative transfer" seen on physical and functional data, where joint models lose to separately trained ones, comes from this mismatch. The fast-growing task's rare extreme inputs dominate the shared gradient, and per-task scalar rebalancing cannot see the problem, because the imbalance sits inside a task, across regions of input space.

The proposal has two parts. The first is a shared feature extractor whose units mix activations of several growth orders, so that each task's linear output layer can pick the growth class it needs. The second is a per-task loss weighted by the task's own growth-matched norm, which divides each sample's error by $1+\rho(x)^{k_t}$, where $\rho(x)$ is the input's size and $k_t$ is task $t$'s growth order. Together these make every task's error comparable across all input scales before any between-task balancing happens. We give a short theory that predicts when this matters, namely heavy-tailed inputs and tasks with different growth orders, and test that prediction against strong, simple alternatives.

Core Claim

Two concrete statements carry the idea. First, an architectural limitation. A shared trunk built from rectified linear units (ReLU, $\max(0,x)$) of any depth produces piecewise-linear features, so every linear head on top of it grows at most linearly. For a task whose target $f_t$ satisfies $|f_t(x)|/(1+|x|^{k})\to c\neq 0$ with $k>1$, the error measured in the weighted norm $\|f\|_{Y_k}=\sup_x |f(x)|/(1+|x|^k)$ is therefore bounded below by $|c|$, however wide the network. Shared ReLU multi-task models are thus structurally unable to serve a superlinear task and a linear task with one backbone in the global sense. A trunk whose units are split among $\text{ReLU}^j$ channels, $j\in\{1,\dots,K\}$, does not have this limitation. Density of such a shared two-layer model in the product space $Y_{k_1}\times\dots\times Y_{k_T}$ (for $k_t\le K$) follows almost directly from Hatano et al.'s Theorem 1.1. We present it as a modest corollary, not a headline result. The open quantity we will study is how the width budget must be split across growth orders when tasks share channels.

Second, an optimization claim, which is the empirically important one. Under squared loss, task $t$'s contribution to the shared gradient from inputs with $\rho(x)>r$ scales roughly like $r^{2k_t}\,P(\rho(x)>r)$, because both the residual and its parameter sensitivity grow like $\rho^{k_t}$. With light (Gaussian) input tails this term vanishes. With polynomial tails it can dominate the trunk's updates for high-$k_t$ tasks. DB-MTL's log-loss and max-gradient-norm rescaling (Dual-balancing for multi-task learning) multiplies a whole task's gradient by one scalar, so the trunk is still shaped mostly by the tail of the fastest-growing task. The growth-matched loss

$$\mathcal{L}_t=\mathbb{E}\left[\frac{(g_t(h(x))-y_t)^2}{(1+\rho(x)^{k_t})^2}\right]$$

is the mean-square analogue of the $Y_{k_t}$ norm. Here $h$ is the shared trunk, $g_t$ is task $t$'s head, and $\rho(x)$ is a problem-specific size measure: input norm, atom count, or the norm of a curve's principal-component scores. This loss removes the tail dominance at its source. Between-task balancing such as DB-MTL can then be applied on top.

The decisive prediction is that the gains should appear only when input tails are heavy and the tasks' growth orders differ. Under Gaussian inputs or equal growth orders, our method should match the baselines. If gains appear regardless, our explanation is wrong even if the method helps.

Relation to Prior Work

Approximation theory supplies the mechanism. Hatano et al. (2020) prove that two-layer ReLU networks are dense on all of $\mathbb{R}$ under the linear-growth weighted norm. The 2024 paper extends this to $\text{ReLU}^k$ and $k$-sigmoidal activations in $Y_k(\mathbb{R})$. Kratsios (2021) obtains global universality on $\mathbb{R}^m$ through growth-controlling wrappers and notes that frozen or shared layers keep universality. All three are single-output, so none asks what happens when one representation must serve targets of different growth orders. That multi-output question is ours. We also inherit their limitation: the density theorems are one-dimensional and two-layer, so for $\mathbb{R}^d$ we use a radial size measure $\rho$ and treat extension as a conjecture checked empirically.

The multi-task literature supplies the baselines and the evidence of failure. RotoGrad homogenizes gradient magnitudes and rotates shared features per task to reduce directional conflict. DB-MTL compresses loss scales with a logarithm and equalizes gradient norms, and it reports that on QM9 no multi-task method beats single-task training. That failure is our main motivation. Both methods act per task and per step, never per region of input space.

Unbounded-input architectures supply alternatives to our mixed trunk. UKAN generates spline coefficients on an unbounded grid and is evaluated on multi-property MoleculeNet tasks, but it analyzes neither shared-gradient dynamics nor growth orders. lmKAN uses a sigmoid-shaped "sigma grid" to cover the whole real line. That design compactifies the input, which is exactly the move we test against.

For functional inputs, Zhou et al. (2026) show that the principal-component scores of curves are unbounded and prove rates for single-task ReLU regression. They explicitly leave multi-task sharing open, and their household-electricity application already pairs a growing regression target with a bounded classification target. A Foundation Model for Virtual Sensors shows that a unified model can beat isolated ones on hundreds of physical signals with very different ranges, but it trains with plain mean squared error and does not study cross-sensor interference.

Research Program

The theory work is deliberately small. We will prove the ReLU lower bound in the multi-output setting and state the mixed-trunk density corollary. We will then derive the tail-contribution scaling of shared gradients under polynomial-tailed input laws, with an explicit condition on the tail index under which the unweighted high-growth task dominates the trunk.

On the empirical side we build three components. The first is the mixed-growth trunk: a standard backbone whose final shared layer's units are partitioned among $\text{ReLU}^1$, $\text{ReLU}^2$, $\text{ReLU}^3$, and a bounded sigmoid channel for classification tasks. The second is a growth-order estimator. It fits the slope of $\log|y_t|$ against $\log\rho(x)$ on the upper tail of the training data, or uses a physical prior where one exists, such as extensive versus intensive molecular properties. The third is a gradient-attribution diagnostic. It records, for each task, what fraction of the trunk's gradient norm comes from the top 5% of inputs by $\rho$. This diagnostic is how we check the mechanism rather than only the outcome.

The benchmarks run from controlled to realistic. On synthetic data, input laws range from Gaussian to Student-t with tail index 3, and tasks have prescribed growth orders 0, 1, 2 and 3. On QM9, $\rho$ is atom count, and we exploit the extensive/intensive split among its 12 targets. On the UK Household Electricity data, $\rho$ is the norm of the curve's principal-component scores, and the tasks are peak regression (growth order about 1) and exceedance classification (growth order 0). The Audi e-tron CAN-bus data serves as a stress test with very heterogeneous sensor ranges.

Baselines

The naive reference point is equal-weight summation of task losses on a shared ReLU trunk, with z-scored inputs and targets. This is what most practitioners run. Alongside it we train single-task models, one network per task with the same total parameter budget split evenly across tasks. On QM9, DB-MTL reports that single-task training beats every multi-task method, so beating single-task training on QM9 is the clearest bar.

The strongest existing multi-task method is DB-MTL, which DB-MTL's own paper reports as best among multi-task methods on QM9 and NYUv2. We also run RotoGrad as the strongest direction-aware competitor, with a block-diagonal rotation where the feature dimension makes full rotation too costly. On the CAN-bus data the strongest reported reference is the unified virtual-sensor model, with an MSE of 0.121 versus 0.177 for isolated per-sensor models (A Foundation Model for Virtual Sensors). We will use its architecture as the backbone there and change only the trunk activations and the loss. For QM9 and NYUv2 the file gives no numeric values: no reported result was found on Lacuna for DB-MTL's per-target QM9 errors or for NYUv2 metrics. For UK Household Electricity, no reported result was found on Lacuna for the error values. We will therefore rerun DB-MTL ourselves in a shared codebase.

Fairness is enforced in three ways. Every method gets the same backbone width, the same data splits and the same number of epochs. Each method gets the same budget of 30 random-search hyperparameter trials per benchmark, and our $k_t$ estimates count against that budget. All results are averaged over five seeds.

Two ablations isolate the key ingredients. The first uses the mixed trunk with unweighted loss. The second uses a ReLU trunk with growth-weighted loss. Both are also run with DB-MTL layered on top, to show whether the weighting and between-task balancing are complementary.

Our method counts as beating a baseline only if three conditions hold. It must improve the worst-case task's error on held-out extreme inputs (above the 95th percentile of $\rho$) by more than seed variance. It must not lose on the bulk of the data. And on QM9 it must close at least half the gap between DB-MTL and single-task training on the extensive targets.

Why not just…?

Why not just standardize, clip, or squash the inputs into a bounded box? Squashing is the main threat. It is what lmKAN's sigma grid effectively does, and it works when test inputs stay inside the training range. The theory says the problem is different: a bounded-input, bounded-activation model cannot follow polynomial growth beyond the training range, and z-scoring changes scale but not tail shape. For curves, the unbounded principal-component scores in Zhou et al. (2026) make clipping a real distortion. We include tanh-squashed and quantile-clipped variants of every baseline. If they match us on the extreme-input split, the architectural half of the proposal is unnecessary, and we will say so.

Why not just transform the targets, for example with a signed logarithm, or divide each target by $\rho^{k_t}$? This is the closest simple competitor. Dividing the target by $\rho^{k_t}$ is nearly equivalent to our weighted loss with a different head parametrization, so we run it as a baseline rather than argue against it. A log transform changes what is being optimized toward relative error. It also flattens polynomial structure, which should hurt extrapolation in original units. We evaluate all methods in original units and in the weighted norm. If target division alone captures the gains, the contribution reduces to a clean explanation of negative transfer plus a recipe, and we will report that honestly.

Why not just use a robust loss such as Huber, which already downweights large errors? Huber downweights samples by the size of their residual, not by their input size. Early in training, bulk samples also have large residuals, so Huber cannot single out the tail. Late in training it treats a large-$\rho$ sample that is fit well like any other. The gradient-attribution diagnostic will show whether Huber reduces tail dominance of the trunk as much as growth weighting does. Huber plus DB-MTL is included as a baseline.

Why not just train separate models? On QM9 that is currently the winning choice. But separate models scale linearly in cost with the number of tasks, and the virtual-sensor results show joint models winning on heterogeneous physical signals (0.121 vs. 0.177 MSE). Our claim is narrower: once growth mismatch is removed, sharing should stop hurting. The QM9 comparison tests this directly.

Decisive Checks

The first check is on the synthetic suite. We cross input tail heaviness (Gaussian versus Student-t) with growth-order disparity (equal versus orders 1 and 3). The method should help only in the heavy-tailed, disparate cell. The diagnostic should show the fast-growing task supplying most of the trunk gradient there under DB-MTL, and much less under growth weighting. If gains appear in the Gaussian or equal-order cells, our mechanism is not the explanation.

The second check is on QM9. Splitting results into extensive and intensive targets should reveal that DB-MTL's deficit relative to single-task training is concentrated in the extensive targets, and that growth weighting removes most of it. Evaluating on the largest molecules, a held-out size shell, makes the extrapolation claim measurable.

The third check is on the household-electricity data. The regression-plus-classification pair should show the bounded classification head being harmed by the regression tail under plain joint training, and recovering under our loss.

A clean negative result would also be informative. If target division or squashing matches us everywhere, we will have shown that growth mismatch explains negative transfer but needs no new architecture.