Low-Rank Stacking: Fusing Many Experts from Few Labels

When several trained classifiers disagree, a good way to combine them is stacking: feed every model's predicted class probabilities into a second, small model (the "meta-learner") that learns whom to trust, and when, from a labeled held-out set. This works well. On a CIFAR-100 benchmark split across 20 specialist models, a learned fuser reaches 82.13% accuracy while plain probability averaging gets 76.64% (Fusion of Experts). The catch is that the meta-learner's input has one entry per expert per class. With $K$ experts and $C$ classes, a linear meta-learner already has $C \times KC$ weights, which is 200,000 for the CIFAR-100 setup, and modern fusers are unregularized multilayer perceptrons on top of that. Stacking therefore depends on a large labeled validation set, and both the original and recent stacking literature name this dependence, together with breakdown at many classes, as an open problem (Ting and Witten 1999, LazyStack).

Our claim is that most of those weights are not needed. The useful information in a stacked ensemble moves through a few channels. The first is a per-expert, per-class competence score: how much to trust expert $k$ when it votes for class $c$. The second is a low-dimensional signal saying which "domain" an input belongs to, which is what lets the fuser silence specialists that are confidently wrong outside their own domain. Both structures can be written as penalties from high-dimensional statistics that split cleanly across subspaces, the "decomposable norms" studied by Negahban et al.. A group penalty over experts captures competence and lets whole experts be dropped. A nuclear-norm penalty, which pushes a matrix toward low rank, captures cross-expert routing. With these penalties the number of labels a meta-learner needs should scale with the true structure (rank times dimension, plus the number of useful experts) rather than with $K \cdot C^2$. We will build this estimator, prove its error rate, and test whether it matches a full neural fuser using a fraction of the validation labels.

Core Claim

Let $z(x) \in \mathbb{R}^{KC}$ stack each expert's log-probability vector, centered within each expert, so that adding a constant to one expert's logits changes nothing. The meta-learner outputs class scores $f(x) = b + (D + L)\,z(x)$. Here $D$ is nonzero only on the entries that connect expert $k$'s vote for class $c$ to output class $c$. Its nonzero entries form one competence vector $d_k \in \mathbb{R}^C$ per expert. $L \in \mathbb{R}^{C \times KC}$ is a dense cross-term matrix that we expect to have low rank, because input-domain evidence is low-dimensional. We fit

$$ \min_{b, D, L}\ \frac{1}{N}\sum_{n=1}^{N} \ell\big(y_n,\ b + (D+L)z(x_n)\big) \;+\; \lambda_L \|L\|_* \;+\; \lambda_D \sum_{k=1}^{K} \|d_k\|_2 , $$

where $\ell$ is softmax cross-entropy, $N$ is the number of labeled validation examples, $\|L\|_*$ is the sum of singular values of $L$, and $\lambda_L, \lambda_D$ are penalty weights. Both penalties are decomposable in the sense of Negahban et al.: the nuclear norm splits across singular subspaces and the group norm splits across expert supports. Their sum is a hybrid penalty of the kind Lee et al. handle as a support function. Suppose $L^*$ has rank $r$ and only $s$ of the $K$ experts carry competence. The master theorem then suggests Frobenius error of order $\sqrt{r(C + KC)/N} + \sqrt{sC\log K/N}$. The first term is the nuclear-norm rate with matrix dimensions $C$ and $KC$. The second is the group-lasso rate. By comparison, an unstructured linear fuser has error of order $\sqrt{KC^2/N}$. The gap between the two predicts where structure should matter most: small $N$, large $C$.

The main difference from prior work is the target. Prior work regularizes ensemble weights as a vector, using sparsity or non-negativity. We impose a matrix structure, competence plus a low-rank route, on the multi-class fuser.

Relation to Prior Work

Two strands of prior work motivate this design. Ting and Witten (1997) and (1999) showed that feeding probabilities to a multi-response linear regression beats feeding labels: 9 of 10 UCI datasets against picking the single best model by cross-validation, and 8 of 10 against majority vote. They also found that linear fusers become unstable on small data and that non-negativity constraints do not change accuracy. Their datasets had low dimension, so they never faced $KC \gg N$, which is the regime we target. Fusion of Experts shows why averaging fails when expertise is complementary: out-of-domain models are confidently wrong. It fixes this with an unregularized MLP fuser and a separate graph-search heuristic (FrugalFoE) to query fewer experts. Our nuclear-norm term targets that same out-of-domain failure with a structure we can analyze, and our group term replaces the pruning heuristic with selection that comes with guarantees. LazyStack reports that confidence-based aggregation degrades on ImageNet-1K. That result constrains us: we must show our gains do not also vanish at $C = 1000$.

The nearest existing estimator is nsslasso (Shen et al. 2024). It stacks pathway-level predictors with a non-negative spike-and-slab lasso and reports AUC 0.750 on TCGA-BRCA using 5 pathways. nsslasso works on scalar base predictions with a vector sparsity prior fitted by EM. It has no matrix structure across classes and no cross-expert terms, and it provides no recovery theory. The collinearity arguments for constrained ensemble weights (Perrone, constrained-weight line; On different ensembles of kernel machines) motivate regularization in general. Ridge is the default there, and it neither selects experts nor finds low rank.

The method and proof tools come from three sources. Negahban et al. supply the cone and restricted-curvature argument. Agarwal et al. supply the low-rank-plus-structured separation and its "spikiness" condition, which we must adapt so that $D$ and $L$ are identifiable. Lee et al. supply the irrepresentable-type condition for exactly recovering which experts matter.

Research Program

The theory has three parts. The first is restricted curvature. Simplex-valued features are bounded and correlated, so we must show that cross-entropy has positive curvature in the directions where the error cone lies. We will prove this under a condition on expert disagreement, roughly that the covariance of centered log-probabilities is nondegenerate on low-rank-plus-group-sparse directions. We will also check the condition empirically on real expert pools by computing restricted eigenvalues of the validation design. The second part is identifiability of $D$ versus $L$, via a spikiness bound on $L$ analogous to Agarwal et al.. The third part is conditions for consistent expert selection, with and without non-negativity on $d_k$. This directly addresses the unresolved disagreement over whether constraining ensemble weights to be non-negative helps. The loss is smooth, so the differentiability requirement in this theory is met.

On the algorithm side, we fit by proximal gradient: singular-value thresholding for $L$ and group soft-thresholding for $D$. For many classes we use a randomized partial SVD. Choosing penalty weights from data is an acknowledged gap in this theory. We will compare cross-validation against a theory-scaled default, $\lambda_L \propto \sqrt{(C + KC)/N}$, and report how sensitive results are to each.

A secondary question is whether the same penalty helps a nonlinear fuser. We apply the nuclear and expert-group penalties to the first-layer weights of a FoE-style MLP with a smooth activation. Liu (2025)'s stationary-point theory then gives a route to error bounds for that nonconvex case.

Baselines

The naive reference is uniform averaging of probabilities, which is parameter-free and needs no labels. On partitioned CIFAR-100 it is reported at 76.64% accuracy (Fusion of Experts). We will also run learned per-expert scalar weights, which are a rank-one special case of our model, to separate "learning weights at all" from "learning structure."

The strongest existing method is the FoE MLP fuser, reported at 82.13% on partitioned CIFAR-100 with 20 experts and 91.88% average accuracy on the five-domain sentiment benchmark (Fusion of Experts). The file does not state the validation size behind these numbers. We will therefore retrain FoE ourselves on identical expert pools and identical validation subsets, and give it the same hyperparameter search budget as our method (weight decay, width, early stopping). We will first confirm that we reproduce roughly 82% at full validation size. Beating FoE means one of two outcomes. Either we reach FoE's full-data accuracy with at most a quarter of its labels, or at every validation size up to 50 labels per class we score higher by a margin outside seed variance, while staying within one point at full data.

The classical linear competitor is multi-response linear or multinomial logistic stacking (Ting and Witten 1999) with a tuned ridge penalty, plus a variant with non-negativity. For expert pruning, the competitor is FrugalFoE, which reaches peak accuracy while querying 37.5% of experts (Fusion of Experts). We compare accuracy at matched expert budgets. We also adapt an nsslasso-style non-negative lasso on per-expert scores (Shen et al. 2024) as the vector-sparsity alternative.

Ablations isolate each ingredient: nuclear term only, competence term only, both, raw probabilities versus centered log-probabilities, and a low-rank-plus-entrywise-sparse variant in place of the expert-grouped $D$.

Why not just…?

Why not just average the probabilities? Averaging needs no labels. It also fails exactly where expertise is complementary: on partitioned CIFAR-100 it trails the learned fuser by 5.49 points because out-of-domain experts are confidently wrong (Fusion of Experts). Our learning curves start at very few labels per class, so they will show where, if anywhere, averaging is still preferable. We will report that crossover point honestly.

Why not just use a ridge-penalized MLP or linear fuser with early stopping? This is the real threat. Ridge shrinks every weight equally and cannot express "few domains, few useful experts." It may still be nearly as good when labels are plentiful. The decisive test is the accuracy-versus-$N$ curve. If tuned ridge or weight-decayed FoE matches us at every validation size on CIFAR-100 and ImageNet-1K, our claim fails, and we will say so. We also report the effective rank of the best ridge solution. If that solution is full-rank, the low-rank premise is wrong for these pools regardless of who wins.

Why not just parameterize $L = UV^\top$ with small inner dimension and train by gradient descent? An explicit factorization may perform just as well, and if so that is a finding about structure, not about the penalty. The nuclear-norm version has three advantages: a convex fit, rank chosen automatically by $\lambda_L$, and the error bound above. We run the factorized model with its rank tuned on the same budget. If it ties, the contribution reduces to identifying the competence-plus-low-rank structure and its sample complexity, which we would still consider worth reporting.

Proposed Experiments

The first experiment is synthetic. We simulate expert pools with known latent domains, varying the rank $r$, $K$, $C$, and $N$, and check that error grows with $r(C + KC)/N$ and not with $KC^2/N$. We also check that expert selection succeeds exactly when the adapted irrepresentable condition holds.

The real-data experiments use partitioned CIFAR-100 with 20 experts and the five-domain sentiment pool from Fusion of Experts. We sweep validation size from 2 to the full set per class and measure top-1 accuracy, calibration error, and experts queried. For the many-class stress test we use ImageNet-1K with a pool of pretrained models; no reported number for stacking was found on Lacuna for this setting, so all comparisons there come from our own runs. For continuity with classical stacking, and to test the small-sample instability Ting and Witten report, we use their 10 UCI datasets, scored as win/tie/loss against multi-response linear regression. MMLU with 15 LLMs is an optional extension; no reported number was found on Lacuna for it.

The decisive check is a single plot per benchmark: accuracy against labeled validation size for averaging, ridge stacking, FoE, factorized low-rank, and our estimator, all with matched tuning budgets. The proposal succeeds if our curve dominates at small $N$ and the gap widens as $C$ grows. It fails if the curves coincide, or if the gap closes on ImageNet-1K the way confidence-based exits do in LazyStack.