State Transfer Across Optimization Regimes

20 concepts11 papers

When we train a neural network or optimize a complex black-box function, we rarely rely on a single static configuration from start to finish. We might scale a model's architecture mid-way through training, switch from an exploratory optimization algorithm to a local exploitative one, or resume a paused training run from a checkpoint.

In all these scenarios, we face a hidden bottleneck: optimization is fundamentally path-dependent. An optimizer does not just track the current position in the parameter space; it accumulates a rich internal "memory" of the landscape. This memory takes the form of momentum vectors, moving averages of gradients, population distributions, or covariance matrices.

If you transition to a new optimization regime—whether that means changing the algorithm, scaling the model, or starting a new training phase—and reset this internal state to zero, you destroy this accumulated context. The system suffers a severe performance penalty, often experiencing massive loss spikes or wasting thousands of iterations just re-learning its trajectory. Optimizer state transfer is the practice of preserving, translating, and injecting this internal history across algorithmic and structural boundaries to maintain optimization momentum.


The Anatomy of Optimizer Memory

To understand why state transfer is necessary, consider what an optimizer actually tracks. In simple Stochastic Gradient Descent (SGD), the state is just the current weights. But modern optimization landscapes are highly non-convex, requiring algorithms to maintain historical context to navigate efficiently:

  • Deep Learning Optimizers (e.g., Adam, RMSprop): Track the first moment (momentum, or the direction of consistent descent) and the second moment (an uncentered variance used to adaptively scale learning rates per parameter).
  • Black-Box and Evolutionary Algorithms (e.g., CMA-ES, Differential Evolution): Track populations of candidate solutions, distribution centers, and covariance matrices that capture the topography of the local search space.

If an algorithm transition occurs without state transfer, the incoming solver starts with a "blank slate." For example, as shown by Vermetten et al. in To Switch or Not to Switch: Predicting the Benefit of Switching Between Algorithms Based on Trajectory Features, a lack of seamless state transfer forces the new solver to expend redundant function evaluations simply re-aligning its internal parameters to the existing search space.

Without State Transfer:
[Solver A (Accumulates Memory)] ──> (Reset State) ──> [Solver B (Starts Blindly from Scratch)]

With State Transfer:
[Solver A (Accumulates Memory)] ──> (Map & Translate State) ──> [Solver B (Retains Momentum)]

Warm-Starting Dynamic Algorithm Selection

One of the most direct applications of state transfer is dynamic algorithm selection in numerical black-box optimization. Different algorithms excel at different phases of a search; an aggressive global search might be ideal early on, while a precise local solver is better near convergence.

In Towards Dynamic Algorithm Selection for Numerical Black-Box Optimization, Vermetten et al. demonstrate that switching between solvers mid-run can yield massive speedups, but emphasize that transferring internal state information—such as population data or covariance matrices—is a critical implementation hurdle. Without it, the cost of re-initializing the search can completely negate the benefits of the switch.

To solve this, frameworks use explicit warm-starting mechanisms. As detailed by Kostovska et al. in Per-run Algorithm Selection with Warm-starting using Trajectory-based Features, when a transition occurs, the search state—including the best-found solutions and distribution centers—is actively mapped to the successor algorithm. This ensures the new solver begins from an informed position, preserving search momentum.

Similarly, in the domain of evolutionary algorithms, Guo et al. developed a reinforcement learning framework (RL-DAS) that dynamically schedules different solvers. They show that saving the internal parameters of an outgoing algorithm and restoring them upon reactivation prevents the total reset of progress, allowing the system to cycle through algorithms seamlessly. This concept of state preservation even extends to data stream processing; Wang et al. demonstrate in MOStream that migrating clustering results smoothly between configurations prevents catastrophic performance drops when underlying data structures shift.


The Cross-Optimizer Challenge in Deep Learning

While transferring a population of points in black-box optimization is relatively straightforward, transferring deep learning optimizer states is significantly more complex. Different optimizers track fundamentally different mathematical quantities.

In Towards Dynamic Interleaving Optimizers, Chen et al. introduce a framework (DOIT) that dynamically switches between different deep learning optimizers (like Adam and SGD) to maximize convergence speed. They note that a primary complication is that each optimizer maintains unique moving averages of gradients. Forcing a hard reset of these states disrupts training stability. To make interleaving work, the historical gradient information must be translated across algorithmic regimes rather than discarded.

This challenge is echoed when attempting to transfer entire training trajectories across different network initializations. Chijiwa notes in Transferring Learning Trajectories of Neural Networks that aligning dynamic internal states like momentum or adaptive learning rates across different initializations is significantly more complex than aligning standard weight updates, presenting a major hurdle for learning transfer.


Preserving Trajectories During Model Scaling and Restarts

State transfer is equally vital when the model architecture changes but the optimizer remains the same. A prominent example is progressive model scaling, where a large model is initialized using a smaller, pretrained counterpart to save computational costs.

Conventional scaling techniques typically copy the model weights but ignore the optimizer's internal states. In TripLe: Revisiting Pretrained Model Reuse and Progressive Learning for Efficient Vision Transformer Scaling and Searching, Fu et al. argue that omitting the accumulated gradient history forces the larger model to completely re-learn the optimization trajectory. By explicitly scaling and transferring the optimizer's internal states (momentum and variance) alongside the weights, the larger model bypasses this costly re-learning phase, slashing training costs by up to 80%.

Standard Weight Scaling:
[Small Model Weights] ──> [Large Model Weights]
[Small Optimizer State] ──> (Discarded) ──> [Large Optimizer State Reset to Zero] (Re-learns Trajectory)

TripLe State Transfer:
[Small Model Weights] ──> [Large Model Weights]
[Small Optimizer State] ──> (Scaled & Mapped) ──> [Large Optimizer State Retained] (Smooth Convergence)

Even without architectural changes, maintaining state continuity is critical across distinct training phases or unexpected restarts:

  • Training Restarts: When training massive language models, runs frequently interrupt due to hardware faults. In a study on training dynamics, Li et al. demonstrate in Training Dynamics of a 1.7B LLaMa Model that failing to restore optimizer states when resuming from checkpoints causes severe validation loss spikes and prolonged recovery periods.
  • Iterative Fine-Tuning: In Chain of LoRA (COLA), Qin et al. construct an iterative residual learning chain by repeatedly training and merging low-rank adaptation modules. They observe that resetting the optimizer state at each step loses valuable historical gradient information, suggesting that advanced momentum transition methods are vital for residual learning efficiency.
  • Self-Evolving Training: For models that improve by learning from their own generated data, Liu et al. introduce M-STAR, a framework that deliberately maintains optimizer states and learning rate schedulers across iterative self-improvement cycles. This persistent state management ensures a smooth transition between data generation and model updates, preventing performance saturation.

Why It Works: The Intuition

Optimizer state transfer works because the geometry of a loss landscape contains directional consistency across scales, steps, and related algorithmic representations. The history accumulated by an optimizer encodes critical information about this geometry: which directions are highly volatile (requiring a smaller effective learning rate) and which directions are consistently steep (allowing momentum to accelerate progress).

When we change the solver or scale the model, the underlying optimization landscape does not instantly become unrecognizable; its general topography remains correlated. By treating the optimizer's internal memory as a first-class citizen and actively translating it to new contexts, we preserve the kinetic energy of the optimization process, ensuring that our models never have to learn how to walk twice.

Key Foundational Work

Per-run Algorithm Selection with Warm-starting using Trajectory-based Features (2022) — contributed 3 concept elements to this direction.
Towards Dynamic Algorithm Selection for Numerical Black-Box Optimization: Investigating BBOB as a Use Case (2020) — contributed 2 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_optimizer_state_transfer_77a0f1bf,
  title = {optimizer state transfer},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/optimizer-state-transfer/rdr_77a0f1bf688744dcb797ac97c6bb7728},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_optimizer_state_transfer_77a0f1bf_v1,
  title = {optimizer state transfer},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/optimizer-state-transfer/rdr_77a0f1bf688744dcb797ac97c6bb7728/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}

See something wrong? Report a mistake