Letting Confounders Explain Themselves in Infinite Mixtures of Inverse Regressions

Many prediction problems map a huge input to a few numbers. Examples include head-pose angles from a 4096-dimensional image feature, or mineral concentrations from a hyperspectral spectrum. One family of methods handles this well by working backwards. Instead of learning the input-to-target map directly, it learns how the few target numbers generate the large input as a mixture of local linear pieces, called "experts". It then flips that model with Bayes' rule to predict. Two separate fixes exist for this family's weaknesses. One adds a small hidden vector that soaks up nuisance variation such as lighting or identity, so that variation does not corrupt the prediction. The other uses a Bayesian nonparametric prior, a prior that lets the data decide how many experts to use instead of fixing that number in advance.

This note argues that the two fixes cannot be applied separately. A mixture that may add experts freely but has no place to put nuisance variation will spend its extra experts explaining lighting. The result is over-fragmentation: many small redundant clusters, a failure the nonparametric literature already documents. We propose a single model that infers both the number of experts and, for each expert separately, how many hidden nuisance dimensions it needs. We then test the specific prediction that this removes spurious experts while improving accuracy on confounded data.

Core Claim

Gaussian Locally-Linear Mapping (GLLiM), introduced in High-dimensional regression with gaussian mixtures and partially-latent response variables, models a high-dimensional observation $X\in\mathbb{R}^D$ as a mixture of $K$ affine functions of a low-dimensional target. Its hybrid variant (hGLLiM) appends an unobserved vector $W\in\mathbb{R}^{d_w}$ to the target. On the Stanford face data, treating lighting this way improved pose accuracy by 20–60%. In hGLLiM both $K$ and $d_w$ are fixed by hand. Robust Head-Pose Estimation Based on Partially-Latent Mixture of Linear Regression picks $K$ from Bayesian Information Criterion (BIC) curves and sets $d_w=4$ for every expert.

The closest existing work is Bayesian nonparametric mixture of experts for inverse problems (BNP-GLLiM). It puts Dirichlet-process and Pitman–Yor priors over the experts and fits them by variational Bayes. It adds a Merge-Truncate-Merge (MTM) post-processing step to collapse redundant clusters and proves posterior consistency for the number of experts. However, it models only fully observed targets, and it is validated on a confounder-free synthetic "π-shape" problem.

Our claim is that, on data with nuisance variation, the redundant experts that MTM must clean up after the fact are largely a symptom of the missing latent term. We further claim that giving each expert an adaptively sized latent subspace fixes this at the model level. The result should be fewer active experts, closer to the true number, together with lower prediction error than either BNP-GLLiM or hand-tuned hGLLiM. This is a falsifiable statement about how two kinds of model capacity trade off, not only a merge of two models.

Model

Each expert $k$ in a stick-breaking (Dirichlet or Pitman–Yor) mixture generates the observation as

$$X = A_k T + B_k W + b_k + E_k,\qquad W\sim\mathcal{N}(0,I_{d_{\max}}),\quad E_k\sim\mathcal{N}(0,\Sigma_k),$$

where the symbols are as follows:

  • $T\in\mathbb{R}^{d_t}$ is the observed target, with an expert-specific Gaussian prior $T\mid k\sim\mathcal{N}(c_k,\Gamma_k)$.
  • $A_k$ and $b_k$ are the expert's affine map and offset.
  • $B_k\in\mathbb{R}^{D\times d_{\max}}$ holds the loadings of the nuisance vector.
  • $\Sigma_k$ is a diagonal noise covariance.

Each column $j$ of $B_k$ receives its own precision $\gamma_{kj}\sim\mathrm{Gamma}(a,b)$. This is automatic relevance determination: columns the data do not support are shrunk to zero. The effective nuisance dimension of expert $k$, $d_{w,k}$, is the number of surviving columns.

Marginalizing $W$ gives each expert a factor-analysis covariance $B_kB_k^\top+\Sigma_k$. The parameter count therefore stays linear in $D$, which is the property that makes GLLiM work when $D\gg d_t$. The forward predictive density $p(T\mid X)$ remains a closed-form Gaussian mixture, as in the GLLiM inversion lemma.

The per-expert dimension is the idea we borrow from DP-space. That work infers both the number of clusters and a subspace dimension per cluster, but it does so without any regression target. Here the subspace is explicitly a nuisance subspace, defined relative to a supervised target.

Inference is mean-field variational Bayes. We extend BNP-GLLiM's updates with Gaussian posteriors over $W$ and Gamma posteriors over $\gamma_{kj}$, and we keep MTM as an optional final step so its contribution can be measured.

There is an identifiability tension. A curved nuisance manifold can be explained either by more experts or by a larger $W$. We study this directly rather than assume it away, through the prior strengths of the stick-breaking concentration and the Gamma hyperparameters.

Research Program

The work has three parts.

Build. We derive and implement the variational algorithm, including the predictive inversion with $W$ marginalized out.

Mechanism. On synthetic data with a known number of experts and known per-expert nuisance dimensions, we map how the inferred expert count and the inferred nuisance dimensions move as the true nuisance dimension grows. The core prediction is specific. BNP-GLLiM's inferred expert count should rise with nuisance dimension even though the true count is fixed. Our model's count should stay flat, and its inferred dimensions should track the truth.

Real data. On confounded benchmarks we check whether the mechanism yields accuracy and cost gains. We also check whether experts actually use different nuisance dimensions; for example, profile-pose experts may need fewer lighting directions than frontal ones. If the experts all end up with the same dimension, the per-expert ingredient is unnecessary and we will say so.

We deliberately leave out end-to-end deep training, sparsity for $N\ll D$, and a consistency proof for the variational approximation. All three are open problems named in the BNP-GLLiM paper, but each would dilute the test of the central claim.

Baselines

Finite regressor (naive reference). Support vector regression on the same fixed features. This is the finite regressor used as a reference in the hGLLiM head-pose papers. It shows how much any mixture-of-inverse-regressions structure buys.

hGLLiM tuned by grid search. The first strong competitor is hGLLiM (High-dimensional regression with gaussian mixtures and partially-latent response variables) with a shared $d_w$. We grid-search $K$ and $d_w$ by BIC. This is the method practitioners actually use, and it already has the latent term we add. To be fair, it gets the same features and the same initialization scheme. Its total wall-clock budget may equal or exceed ours; we report both its full-grid cost and its accuracy when its compute is capped at our single-run cost. The reported reference figure on Biwi Kinect is yaw MAE 6.06° and pitch MAE 7.65° for hGLLiM_pose&bb-4 (Robust Head-Pose Estimation Based on Partially-Latent Mixture of Linear Regression). On Prima with bounding-box noise, the same paper reports roughly 8°–12° MAE.

BNP-GLLiM. The second strong competitor is BNP-GLLiM with MTM (Bayesian nonparametric mixture of experts for inverse problems), run on identical features with the same truncation level and concentration prior. It shares our nonparametric machinery but has no latent term. No reported result was found on Lacuna for BNP-GLLiM on any real dataset; its only reported result is recovering $K=3$ as the posterior mode in 100 π-shape trials.

DMLIR (external reference). Deep Mixture of Linear Inverse Regressions fine-tunes VGG-16 jointly with a fixed-$K$ inverse mixture. It reports a Biwi MAE of 3.62°, against 4.34° for a fine-tuned fully connected head. Because DMLIR trains its backbone and we use frozen features, we treat 3.62° as an external reference rather than a head-to-head comparison. Our controlled comparison against fixed-$K$ GLLiM is on DMLIR's own frozen VGG features.

Ablations. Two ablations isolate the key ingredients. The first ties all $\gamma_{kj}$ across experts, so all experts share one learned nuisance dimension; this tests the per-expert ingredient. The second removes MTM; this tests whether the latent term makes post-processing largely unnecessary.

What counts as winning. Beating the baselines means two things together. First, lower MAE than grid-searched hGLLiM at equal compute, and no worse at unlimited compute. Second, lower MAE and an expert count closer to the truth than BNP-GLLiM on confounded data.

Why not just…?

Why not just grid-search hGLLiM over $K$ and $d_w$ with BIC? This is the real threat on accuracy. With enough compute it may match us on benchmarks where one shared nuisance dimension suffices. Our advantage claims are narrower. The grid costs roughly $K_{\max}\times d_{\max}$ separate fits, against one fit for our model. A grid also cannot express heterogeneous per-expert dimensions, since that space grows as $d_{\max}^K$. The capped-compute comparison and the tied-precision ablation settle both points. If the tied ablation matches the full model and the uncapped grid matches both, then the contribution reduces to a cost saving, and we will report it that way.

Why not just run BNP-GLLiM with more experts and let them absorb the nuisance? Locally, extra experts can mimic a latent subspace. However, each extra expert pays a full $D$-dimensional offset and noise covariance and fragments the training data. The 20–60% gain from modeling lighting as latent in High-dimensional regression with gaussian mixtures and partially-latent response variables suggests that fragmentation is a poor substitute. The synthetic sweep of expert count against nuisance dimension tests this directly.

Why not just fit a finite hGLLiM with a large $K$ and sparse mixture weights plus automatic relevance determination? This is an "overfitted finite mixture" approach that would empty unneeded components. It differs from our model mainly in the prior on the weights. We include it as an extra ablation, with the same $K_{\max}$ as our truncation level. If it matches us, the finding still stands: the latent nuisance term is what controls fragmentation. The nonparametric prior would then matter mainly for the consistency guarantees inherited from BNP-GLLiM.

Why not just fine-tune a deep network with a direct regression head? On Biwi, DMLIR's fixed-$K$ inverse layer already beats a fine-tuned fully connected head (3.62° versus 4.34°) and stays competitive with 20% of the training data (Deep Mixture of Linear Inverse Regressions). We repeat the 20% subsampling to check that our model keeps this sample efficiency.

Proposed Experiments

Synthetic tests. We extend the π-shape problem from BNP-GLLiM by adding nuisance factors of known per-expert dimension (for example 0, 2 and 5 across the three experts) at increasing strength. We use the same 100-trial protocol. The decisive check is the relationship between nuisance dimension and inferred expert count. The claim holds if BNP-GLLiM's count rises above 3 while ours stays at 3 and recovers the per-expert dimensions. The claim fails if both inflate or both stay at 3. We also run the 50-dimensional nonlinear synthetic task from the GLLiM paper.

Real benchmarks. Real-data tests use four datasets:

  • Stanford faces, where lighting is the known confounder.
  • Prima, with injected bounding-box noise.
  • Biwi Kinect, on frozen VGG features and with 20% subsampling.
  • OMEGA Mars spectra. No absolute error number is reported on Lacuna for OMEGA; the only reported result is GLLiM's 36% lower error than relevance vector machines. We therefore rerun GLLiM there.

Metrics. We report per-angle MAE, predictive negative log-likelihood, the number of active experts before and after MTM, the inferred dimensions, and wall-clock time against the BIC grid.

Secondary check. We examine whether the learned nuisance subspaces align with measured lighting direction on Stanford faces. Alignment would show that the latent term absorbs the intended factor and not target signal.