Variable Projection for Separable Nonlinear Optimization

You are reading version 1 of this direction, frozen 2026-10-04. The latest version is version 2.

Version 1 of 215 papers20 conceptsfrozen 2026-10-04

In many complex optimization problems, from training neural networks to reconstructing 3D scenes, the variables can be split into two distinct groups: those that appear linearly and those that appear nonlinearly. Standard practice is to optimize all variables simultaneously using gradient descent. However, a powerful alternative known as Variable Projection (VarPro) suggests that we should "eliminate" the linear variables entirely, transforming the problem into a lower-dimensional space that is significantly easier to navigate.

The Core Intuition: Conditional Optimality

Imagine you are fitting a model that is a sum of exponentials, a common task in signal processing and Dynamic Mode Decomposition (DMD). The model has "amplitudes" (linear weights) and "decay rates" (nonlinear exponents). If you knew the correct decay rates, finding the optimal amplitudes would be a simple, one-step linear least-squares problem.

Variable Projection exploits this observation. As Christopher Zach, Je Hyeong Hong, and Andrew W Fitzgibbon explain in Revisiting the Variable Projection Method for Separable Nonlinear Least Squares Problems, we can mathematically express the optimal linear parameters as a function of the nonlinear ones. By plugging this expression back into the original objective, we create a "reduced" objective function that depends only on the nonlinear variables.

This means that at every single step of the optimization, the linear weights are guaranteed to be the best possible weights for the current nonlinear features. This "conditional optimality" ensures the optimizer never wastes time exploring regions of the parameter space where the linear weights are poorly matched to the features.

Reshaping the Optimization Landscape

The primary advantage of VarPro isn't just that it reduces the number of variables; it's that it fundamentally changes the geometry of the problem.

In A Saddle Point Remedy: Power of Variable Elimination in Non-convex Optimization, Guang-Yong Chen and Min Gan provide a geometric proof for why this works. They show that eliminating variables through projection can transform "trapping" saddle points—which often stall standard gradient descent—into "repelling" local maxima. By using the Schur complement of the Hessian matrix, they demonstrate that the reduced landscape is smoother and better conditioned.

This smoothing effect is particularly visible in Computer Vision. Dennis Strelow notes in General and Nested Wiberg Minimization that this mechanism allows for much more robust solutions from poor initializations. For instance, in pOSE: Pseudo Object Space Error for Initialization-Free Bundle Adjustment, Christopher Zach and Je Hyeong Hong use VarPro to solve 3D reconstruction problems from completely random starting points, a feat that usually causes standard algorithms to fail immediately.

From Neural Networks to Physics

While VarPro was historically a niche technique for matrix factorization, it is increasingly being applied to modern high-dimensional problems:

  • Deep Learning: Most neural networks end with a linear layer. Elizabeth Newman, Joseph Hart, and colleagues introduced GNvpro, a Gauss-Newton implementation that eliminates these final weights. This leads to faster convergence and better generalization because the network's features are forced to be useful for an optimal classifier at every training step.
  • Scientific Computing: In LieSolver, René P. Klausen and team use VarPro to solve Partial Differential Equations (PDEs). By decoupling the linear weights of basis functions from nonlinear symmetry parameters, they achieve solutions that are 10–100x faster than standard physics-informed neural networks.
  • Dynamic Systems: For analyzing noisy time-series data, Travis Askham demonstrated in Variable projection methods for an optimized dynamic mode decomposition that VarPro significantly reduces the bias in eigenvalue estimation, making DMD much more reliable for real-world experimental data, such as the plasma forecasting studied by Farbod Faraji and Aaron Knoll in Dynamic Mode Decomposition for data-driven analysis and reduced-order modelling of ExB plasmas.

Theoretical Convergence

Recent theoretical work has begun to quantify these gains. In Ultra-fast feature learning for the training of two-layer neural networks in the two-timescale regime, Gabriel Peyré and François-Xavier Vialard prove that when linear weights are updated much faster than nonlinear features (the "two-timescale" regime), the training dynamics follow a predictable diffusion equation. This allows for provable linear convergence rates that are often unattainable with standard joint optimization.

By recognizing the separable structure of a problem, researchers are moving away from "black-box" optimization and toward strategies that analytically simplify the search space, resulting in algorithms that are not only faster but fundamentally more robust to noise and poor initialization.

Papers in this version

History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 12 papers
  2. Version 1 · 2026-10-04 · 15 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_variable_projection_1ecd2e32,
  title = {variable projection},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/variable-projection/rdr_1ecd2e320d1e4e2fa5553a56f0c12afe},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_variable_projection_for_separable_nonlin_1ecd2e32_v1,
  title = {Variable Projection for Separable Nonlinear Optimization},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/variable-projection-for-separable-nonlinear-optimization/rdr_1ecd2e320d1e4e2fa5553a56f0c12afe/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}