High-Dimensional Optimization via One-Dimensional Reduction

65 concepts55 papers1 proposal

High-dimensional optimization is often plagued by the "curse of dimensionality," where the volume of the search space grows exponentially with each added parameter. Standard gradient-based methods can struggle with computational overhead, local minima, or the lack of explicit gradients in "black-box" scenarios. A recurring strategy across recent research is the transformation of these complex, multi-dimensional problems into one-dimensional sub-problems. By reducing the search space to a single dimension—whether through sequential coordinate updates, structural root-finding, or projective mappings—researchers can achieve faster convergence, global optimality, and significantly lower computational costs.

Sequential Coordinate and Block Decomposition

The most direct way to bypass high-dimensional complexity is to optimize one parameter at a time. This coordinate-wise approach simplifies the objective landscape, allowing for more aggressive search strategies in each direction. Wakabayashi et al. demonstrated this in the context of thin-film growth, where sequential one-dimensional optimization navigated high-dimensional parameter spaces to reach record purity levels in just 24 experimental runs.

While simple, the efficiency of this approach depends on the selection rule. Nutini et al. showed that "greedy" selection rules, like the Gauss-Southwell rule, can converge significantly faster than random selection by leveraging the specific geometry of the objective along a single dimension. For global optimization, Tao et al. introduced the ABCD algorithm, which breaks high-dimensional problems into coordinate-wise sub-problems to prevent the exponential growth of function evaluations required by traditional partitioning methods.

Structural Reduction to Root-Finding

In many statistical and geometric models, the high-dimensional update equation possesses a mathematical structure that can be collapsed into a single scalar bottleneck. Instead of solving a massive system of equations, the problem is reduced to finding the root of a one-dimensional monotonic function.

  • Generalized Linear Models (GLMs): Toulis et al. proved that implicit updates for GLMs can be reduced to a one-dimensional root-finding problem for a single scalar, providing stability without the overhead of high-dimensional implicit systems. Similarly, Erdogdu et al. introduced the Scaled Least Squares estimator, which solves for a single scaling constant after an initial linear estimation, achieving maximum-likelihood accuracy at a fraction of the cost.
  • Matrix and Norm Projections: Won showed that the proximity operator for the matrix perspective function—typically a complex semidefinite programming problem—can be solved as a univariate root-finding task. In feature selection, Sra leveraged convex duality to transform projections onto $\ell_{1,q}$-norm balls into a simple one-dimensional search for a dual variable.

Finding Global Optima via 1D Trajectories

One-dimensional reduction is not just a speed optimization; it can be a tool for robustness. In non-convex landscapes where gradient descent might stall in local minima, identifying a specific one-dimensional path that contains the global optimum can guarantee success.

Dekel et al. applied this to hand-eye calibration, reducing a six-degree-of-freedom non-linear optimization to a 1D line search along an eigenvalue curve, ensuring a global least-squares solution. In neural dynamics, Wang et al. reformulated non-convex parameter identification into a search over a single scalar variable, improving reliability in noisy systems. Even in theoretical analysis, Wang et al. used a path-based reparameterization to identify a 1D trajectory containing the optimal solution, allowing them to derive tighter performance bounds for noisy linear regression.

Projective and Human-in-the-Loop Optimization

When the objective function is a "black box" or requires human feedback, high-dimensional queries become cognitively or computationally prohibitive. Reduction techniques project these problems into lower-dimensional subspaces to make them manageable.

Mikkola et al. introduced Projective Preferential Bayesian Optimization, which asks users to optimize along a 1D slider rather than making pairwise comparisons. This provides denser feedback and scales preference learning to high dimensions. For autonomous systems, Cosner et al. restricted Bayesian optimization to one-dimensional lines to enable real-time safety-critical control tuning on quadruped robots.

Closed-Form Solutions via Eigen-Decomposition

Finally, the process of dimensionality reduction itself is often framed as an optimization problem that admits a closed-form solution. By formulating objectives like dependence maximization or local discriminative analysis as generalized eigenvalue problems, researchers avoid iterative optimization entirely. Zhang & Zhou and Sugiyama et al. both utilized this framework to provide globally optimal, non-iterative transformations for multi-label and semi-supervised data, respectively. As Parrish noted, reducing supervised optimization to a single maximal eigenvalue decomposition allows these methods to scale to datasets with tens of thousands of features where iterative methods would falter.

Key Foundational Work

Learning Technique and the Stochastic Approximation Method (1969) — contributed 1 concept elements to this direction.
Some Experiments in Applying Inductive Inference Principles to Surface Reconstruction (1989) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_dimensionality_reduction_for_optimizatio_f12e7118,
  title = {dimensionality reduction for optimization},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/dimensionality-reduction-for-optimization/rdr_f12e71182e214c6cba8b528b442c094b},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_dimensionality_reduction_for_optimizatio_f12e7118_v1,
  title = {dimensionality reduction for optimization},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/dimensionality-reduction-for-optimization/rdr_f12e71182e214c6cba8b528b442c094b/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake