Hybrid First and Second Order Optimization

370 concepts246 papers2 proposals

Training complex models often presents a fundamental trade-off between exploration and precision. First-order methods like Adam or Stochastic Gradient Descent (SGD) are the workhorses of deep learning because they are computationally efficient and robust to the "rugged" landscapes of non-convex functions. However, they frequently stall or saturate as they approach a minimum, failing to achieve the high-precision convergence required for scientific applications. Conversely, second-order methods like L-BFGS or Newton’s method utilize curvature information to achieve superlinear convergence, but they are computationally expensive and highly sensitive to poor initialization.

To resolve this, researchers have converged on a hybrid optimization strategy: a two-phase approach that uses first-order methods to reach the neighborhood of a solution and second-order methods to polish it to high accuracy.

The Two-Phase Regime

The intuition behind hybrid optimization is that the loss landscape changes its character during training. As Leclerc et al. argue, deep network training consists of a "large-step" phase that drives generalization and a "small-step" phase for fine-grained optimization.

In the first phase, the goal is global exploration. First-order methods navigate the initial landscape to find a stable region. For example, in the context of Physics-Informed Neural Networks (PINNs), Rathore et al. and Grossmann et al. show that starting with Adam allows the model to escape poor local minima and reach a basin where the physical residuals are roughly satisfied.

In the second phase, the strategy shifts to local refinement. Once the optimizer reaches the "neighborhood" of a solution, it switches to a second-order method. Jiang et al. introduce the FUSE framework, which formalizes this transition to overcome the saturation problem where first-order methods stall near stationary points. This refinement is critical for high-precision tasks; Wong et al. demonstrate that this hybrid approach achieves mean relative errors below 1.22% in complex engineering simulations, a level of accuracy difficult to reach with gradient descent alone.

The most critical hyperparameter in these strategies is the timing of the switch. Wang et al. found that the scheduled transition from a global stochastic optimizer to a local second-order optimizer is a primary factor in model stability.

Different fields employ varying mechanisms for this transition:

  • Sequential Switching: Qiu et al. and Li et al. use a simple linear handover, where a fixed number of first-order iterations provide a "warm start" for a Newton-based phase.
  • Cyclic Schedules: Fazliani et al. propose alternating between the two, using first-order steps to navigate rugged regions and second-order steps to accelerate whenever a promising basin is identified.
  • Adaptive Sampling: In PINNs, He Wang suggests interspersing the optimization phases with periodic resampling of collocation points, ensuring the high-precision optimizer focuses on the most challenging areas of the physical domain.

Stabilizing Heuristics and Curvature Grafting

Beyond simple switching, some researchers seek to integrate the benefits of both regimes into a single update rule. Clarke et al. propose AdamQLR, which grafts stabilizing heuristics typically reserved for second-order methods—such as automatic damping and trust-region step-size selection—directly onto the Adam optimizer. This approach aims to capture the robustness of second-order methods without the massive computational overhead of explicit Hessian inversion.

Similarly, Mozaffari et al. introduce MKOR, which uses rank-1 updates to make second-order information computationally feasible for large models like BERT. They observe a reverse hybrid pattern: second-order information is most useful for rapid convergence in the initial phases, while transitioning to first-order methods later can maintain efficiency as the model nears a local minimum.

Applications Across Domains

While particularly dominant in scientific machine learning, the hybrid strategy appears across diverse optimization challenges:

  • Tensor Decomposition: Ge et al. combine global Sum-of-Squares algorithms with local search to decompose overcomplete tensors.
  • Federated Learning: Legate et al. use a first-order "warm-up" on high-resource clients before switching to zeroth-order optimization for memory-constrained edge devices.
  • Sparse PCA: Volodymyr Kuleshov combines a first-order power method to find sparsity patterns with a second-order iteration to refine coefficients.

By treating optimization as a multi-stage journey rather than a single descent, these hybrid methods leverage the strengths of different mathematical orders to solve problems that are otherwise computationally intractable or numerically unstable.

History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 253 papers
  2. Version 1 · 2026-10-04 · 310 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_hybrid_first_second_order_optimization_6a8d42e8,
  title = {hybrid first-second order optimization},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/hybrid-first-second-order-optimization/rdr_6a8d42e874d94e4a833b135309ec1c63},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-21},
}

This exact version (version 2, frozen text):

@misc{lacuna_hybrid_first_second_order_optimization_6a8d42e8_v2,
  title = {hybrid first-second order optimization},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/hybrid-first-second-order-optimization/rdr_6a8d42e874d94e4a833b135309ec1c63/v/2},
  note = {Version 2, frozen 2026-10-04. Accessed 2026-11-21},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake