Profiling Out the Linear Part: Near-Free Hyperparameter Tuning for Semiparametric Networks

Many prediction problems have a natural split. Some inputs, such as price, dose, or a known physical baseline, should enter the prediction as a straight line with a coefficient we want to read off and test. Other inputs call for a flexible neural network. These semiparametric models ($y = \beta^T X + g(Z)$, a linear term in covariates $X$ plus a network $g$ on covariates $Z$) are usually trained by pushing $\beta$ and the network weights through the same gradient descent. Their hyperparameters are then tuned by an outer search that repeatedly retrains everything. That search covers the network's learning rate and dropout, and also the penalties on the linear coefficients.

This note proposes a different treatment. The input matrix $X$ never changes during training, so the linear coefficients have an exact ridge-regression solution given the network's current output, and the matrix that solution requires can be factored once and reused. The linear part can therefore be "profiled out", meaning solved analytically at every refresh instead of learned by gradient steps. Its penalty hyperparameters, one per linear feature, then get exact validation gradients at the cost of a small linear solve. The outer hyperparameter search has to cover only the network's own settings.

The claim to test is concrete. On tabular problems with a meaningful linear component, a profiled semiparametric network tuned by a small outer search over the network's hyperparameters only should reach the same held-out loss as a jointly trained hybrid whose full hyperparameter space is searched by a multi-fidelity optimizer. It should get there with several times fewer training runs, and it should recover linear coefficients that are less biased when $X$ and $Z$ are correlated.

Core Claim

Write the training objective with a diagonal ridge penalty $\Lambda = \mathrm{diag}(\lambda_1,\dots,\lambda_p)$ on the $p$ linear coefficients and network parameters $\theta$:

$$\min_\theta \min_\beta \; \frac{1}{n}\,\|y - X\beta - g_\theta(Z)\|^2 + \beta^T \Lambda \beta, \qquad \beta^*(\theta,\Lambda) = (X^T X + n\Lambda)^{-1} X^T\big(y - g_\theta(Z)\big).$$

Three consequences follow.

First, the network is trained on the profiled loss. By the envelope theorem (the derivative of an optimized objective equals its partial derivative at the optimum), the gradient with respect to $\theta$ needs no differentiation through the solve. The profiled loss equals the part of the residual that $X$ cannot explain. The network is therefore implicitly trained on targets that have had the linear projection removed. This is the classical double-residual idea for partially linear models, obtained here without a separate first stage.

Second, the dependence of $\beta^*$ on each penalty is exact: $\partial \beta^*/\partial \lambda_j = -n\,(X^T X + n\Lambda)^{-1} e_j\,\beta^*_j$, where $e_j$ is the $j$-th unit vector. The validation-loss gradient with respect to every $\lambda_j$ therefore costs one extra solve against an already-factored $p \times p$ matrix. It needs no identity-Hessian approximation and no learned best-response model.

Third, per-feature penalties driven by these hypergradients act as automatic relevance determination on the linear side: penalties on useless linear features grow large. This lets us put every candidate covariate in $X$ and every covariate in $Z$, which partly removes the need to decide in advance which features are linear.

The novelty lies in the combination. The linear block is made of fixed raw covariates rather than learned features, which is what makes one-time factorization and exact penalty hypergradients cheap. That exactness is then used to take the linear-side hyperparameters out of the outer search.

Relation to Prior Work

The closest work is Closed-Form Last Layer Optimization. It solves a network's final linear layer by ridge regression during training and uses the envelope theorem to skip backpropagation through the solve. It adds a proximal, Kalman-like term because the features under that layer keep moving. We reuse its envelope-theorem argument but apply it to a different object. Our linear block acts on fixed input covariates, so its Gram matrix $X^T X$ is constant and only the vector $X^T(y - g)$ drifts. That paper also leaves the ridge strength as an ordinary hyperparameter. Our contribution is to tune per-coordinate ridge strengths exactly inside the run.

Two-Timescale Networks for Nonlinear Value Function Approximation motivates the proposal. It pairs a slow nonlinear representation with a fast least-squares linear head in reinforcement learning and finds the linear solver much less sensitive to hyperparameters. We test whether that insensitivity transfers to supervised semiparametric regression and classification, and whether it also reduces the network's own sensitivity to its learning rate.

Two single-run tuning methods supply the comparison point for the hypergradient. The $T_1$–$T_2$ method in Scalable Gradient-Based Tuning of Continuous Regularization Hyperparameters tunes regularizers on a validation split, using greedy updates and an identity-Hessian approximation, at about 30% overhead. Self-Tuning Networks learn an approximate best-response of the weights to the hyperparameters through gating, which doubles parameters. Both approximate what, on the linear block, we can compute exactly. Neither exploits linear structure in the task.

On the modeling side, Sparse Partially Linear Additive Models motivates the proposal. It uses a convex hierarchical group penalty to decide which features are linear, but leaves its trade-off and sparsity parameters to cross-validation. It reports that on RCV1 only 4 of more than 5,000 selected features were nonlinear, which suggests large linear components are common. Neural Additive Models fit a subnetwork to every feature. That creates the opposite problem: many regularizers across many subnets, and gradient descent fitting simple slopes.

LMMNN constrains our design in a useful way. It shows that integrating a linear random-effects block into network training beats treating the two parts separately. Airbnb test MSE was 0.142 for LMMNN against 0.158 for entity embeddings. Its variance components are still learned by plain gradient descent, which is the joint-training regime we want to replace.

Research Program

The system is built in three layers.

The profiled trainer. It caches a factorization of $X^T X + n\Lambda$. It keeps a running estimate of $X^T g_\theta(Z)$ and refreshes it exactly on a full pass every $k$ steps, with stale minibatch corrections in between. It re-solves for $\beta^*$ at each refresh. The refresh period $k$ is the main approximation. Between refreshes the envelope argument holds only approximately, and we will measure how final loss and the accuracy of $\beta$ degrade as $k$ grows. The proximal correction from the closed-form last-layer paper is the fallback if staleness hurts.

For classification, squared loss does not apply. We replace the solve with a few iteratively reweighted least-squares steps, warm-started from the previous $\beta$, so that each refresh is a Newton step on the logistic profile. That paper found squared-loss heads trail cross-entropy on ImageNet, so the logistic extension is necessary rather than optional.

The hypergradient loop on $\Lambda$. It uses either a held-out split or, to avoid spending data, the closed-form leave-one-out residuals of ridge regression conditional on the current $g$. Changing $\Lambda$ requires refactoring at $O(p^3)$ cost, which is acceptable for $p$ up to a few thousand. For wide sparse inputs such as RCV1 we switch to conjugate-gradient solves.

The outer search. An unchanged BOHB (BOHB) searches only the network's hyperparameters: learning rate, width, depth, dropout, and weight decay on $\theta$.

The coefficient analysis. We ask what $\beta$ means when a feature appears in both $X$ and $Z$. Profiling makes $\beta$ the coefficient of the best linear projection with the network fit to what remains. It equals a structural effect only if $g$ carries no linear signal in $X$. We will measure this on semi-synthetic data with known $\beta$ and state plainly where the interpretation holds.

Baselines

The naive reference is a ridge regression on $X$ alone, with its penalty chosen by generalized cross-validation, next to an MLP on all features tuned by BOHB. This reference answers two separate questions: is there any linear signal, and is there any nonlinear signal. Any hybrid that fails to beat the better of the two on held-out loss has no reason to exist on that dataset.

The strongest competitor on the tuning-cost question is a jointly SGD-trained hybrid with the same architecture. It has a linear block and the same $g$, and its full hyperparameter space, including a linear-side ridge strength and a separate linear-block learning rate, is searched by BOHB. This is exactly the practice we claim to improve.

Both methods get the same data splits, the same network search space for $\theta$, the same BOHB settings and fidelity schedule (epochs), and the same total GPU-hour budget, with five or more seeds. Beating it means reaching the competitor's final median test loss with at least 3× fewer total training epochs across the search, and not losing on final loss at equal budget. We found no reported result on Lacuna for BOHB on a semiparametric hybrid of this kind, so this number must be produced here.

On the tabular interpretability sets, we include Neural Additive Models, tuned with the same BOHB budget. Their paper reports MIMIC-II AUC of 0.830, against 0.844 for XGBoost (Neural Additive Models). We report XGBoost as the accuracy ceiling rather than a target. We aim to beat NAM's AUC while keeping exact linear coefficients.

On Airbnb, the reference is LMMNN's reported test MSE of 0.142 (LMMNN). Our profiled trainer adds the host random effect as a ridge-penalized linear block, so the per-level penalties are tuned by our hypergradient.

On Gisette, the reference is SPLAM's reported mean error of 2.18 against 2.43 for Lasso (Sparse Partially Linear Additive Models).

Three ablations isolate the key ingredient.

  • Profiled $\beta$ with a single global $\lambda$ fixed by BOHB. This isolates closed-form solving from exact hypergradients.
  • An SGD-trained $\beta$ whose per-feature penalties are tuned by the $T_1$–$T_2$ approximate hypergradient. This isolates exactness from the mere presence of per-feature tuning.
  • The full method with a refresh period $k$ of one epoch versus ten.

Why not just…?

Why not just use a plain MLP and let BOHB tune weight decay? An overparameterized network can represent a linear function, and on accuracy alone this is the real threat. On some OpenML datasets it may tie. SPLAM's Gisette result (2.18 versus 2.62 for the fully nonlinear SpAM) suggests that fitting linear structure flexibly costs accuracy. The stronger arguments are cost and coefficients. The MLP gives no $\beta$ to test. We also report anytime curves of held-out loss against cumulative training epochs, so a tie in final accuracy but a gap in the compute needed to reach it will be visible. If the MLP matches both, the proposal fails on that dataset, and we will say so.

Why not just fit ridge first and train a network on its residuals? Two-stage fitting has zero linear-side tuning cost. Its flaw is that when $X$ and $Z$ are correlated, the first stage absorbs nonlinear signal into $\beta$ and never revisits it. LMMNN's gap over disjoint baselines is indirect evidence (CelebA error 1.54 versus 3.6 for embeddings). The direct test is our semi-synthetic experiment with controlled correlation between $X$ and $Z$. There, the two-stage estimator's $\beta$ bias should grow with that correlation while the profiled estimator's should not. If two-stage matches profiling at low correlation, we will report that regime as one where the simpler method suffices.

Why not just run $T_1$–$T_2$ hypergradients over the whole jointly trained hybrid? This is exactly the second ablation. That method's identity-Hessian approximation is reasonable for batch-normalized networks, but on a linear block with correlated covariates the true Hessian $X^T X$ is far from identity. Its greedy updates should then mis-set per-feature penalties. We will measure the distance between its $\Lambda$ trajectory and the exact one, and report the resulting difference in test loss.

Why not just pick the linear penalty by generalized cross-validation after training? This is cheap and classical, and it is close to what we do. The difference is that we update the penalties during training, so the network trains against a well-regularized $\beta$ throughout. A post-hoc variant with the network trained at a fixed $\lambda$ is included. If it matches, the in-run loop is unnecessary and the contribution reduces to profiling itself.

Proposed Experiments

Semi-synthetic identification. Draw $X$ and $Z$ from real OpenML tabular covariates, set a known $\beta$, generate a nonlinear $g$, and vary the correlation between $X$ and $Z$. Metrics are $\beta$ bias, bootstrap interval coverage, and test MSE across all methods.

Tuning-cost study. Run on the tabular OpenML datasets used by LCBench and by HPOBench's MLP family, in HPOBench raw mode (HPOBench), because we need fresh training runs rather than lookup tables. Each dataset is treated semiparametrically, with numeric covariates in $X$ and all covariates in $Z$. HPOBench reports no per-optimizer regret numbers on Lacuna, so all regret curves are generated here. We also count how many BOHB evaluations each method needs to reach the joint hybrid's final loss.

Domain datasets. MIMIC-II and COMPAS test interpretable classification with the logistic profile. Airbnb tests a high-cardinality linear block. Gisette and RCV1 test wide linear blocks with conjugate-gradient solves.

Decisive checks. The proposal succeeds if three things hold together:

  • the network's final loss varies less across its learning-rate grid under profiling than under joint SGD, the transfer of the two-timescale finding;
  • at matched final test loss, profiling needs at least 3× fewer total search epochs than the jointly trained BOHB hybrid on a majority of datasets;
  • $\beta$ bias stays flat as the correlation between $X$ and $Z$ rises while two-stage bias grows.

If the first two fail but the third holds, the result is a better estimator, not an answer to tuning overhead, and we will report it that way.