Traditional optimization algorithms are often built on a foundation of constancy. Whether it is the step size in gradient descent or the penalty parameter in a primal-dual method, theory usually dictates a fixed value or a simple, predefined decay schedule. While these "stationary" parameters provide stability and convergence guarantees, they often force a compromise: a parameter that is safe for the final stages of refinement is often sluggish during the initial search.
A growing body of research suggests that we can break this compromise by treating optimization as a sequence of distinct stages, each requiring its own unique parameters. This approach, known as non-stationary parameter scheduling or weight untying in the context of deep unfolding, replaces fixed constants with iteration-specific sequences that adapt to the algorithm's progress.
The Problem with the Average Iterate
In classical convex optimization, many first-order methods only achieve their theoretical optimal convergence rates when you average all previous iterations (the "ergodic" sequence). In practice, however, practitioners almost always prefer the last iterate—the very last point the algorithm reached—because it is usually more accurate and less "blurry" than an average of past states.
Yuzixuan Zhu and Deyi Liu have demonstrated that by using non-stationary parameters, we can bridge this gap. In Non-Stationary First-Order Primal-Dual Algorithms with Faster Convergence Rates, Zhu shows that dynamic sequences of step sizes allow algorithms to achieve optimal convergence rates specifically for the last iterate. This provides a theoretical justification for what practitioners have been doing intuitively: throwing away the history and trusting the final update.
Deep Unfolding: From Heuristics to Learning
While Zhu and Liu derive these parameter sequences through mathematical analysis, another branch of research "learns" them from data. This is the core of deep unfolding.
In deep unfolding, an iterative algorithm (like ADMM or EM) is "unrolled" into a neural network where each iteration becomes a layer. Instead of using a single heuristic constant across all layers, researchers like Alexios Balatsoukas-Stimming "untie" the weights. As noted in Deep Unfolding for Communications Systems, assigning unique, learnable parameters to each iteration allows the system to discover stage-specific optimal values. This can reduce the number of iterations required to reach a target error rate by orders of magnitude.
For example, in distributed optimization, Unfolded D-ADMM uses iteration-specific weights to achieve the same performance as traditional methods but with 8 to 154 times fewer communication rounds.
The Intuition of Stage-Specific Behavior
Why does varying the parameters at every step work so much better? The intuition lies in the changing needs of the optimization process:
- Early Stages (Exploration): The algorithm needs to make aggressive, global adjustments to move from a random starting point toward the general vicinity of the solution. High step sizes or loose penalty parameters facilitate this broad exploration.
- Late Stages (Refinement): As the algorithm nears the optimum, aggressive moves become counterproductive, causing oscillation. The parameters must shift to prioritize "surgical refinement" and local stability.
This transition is explicitly captured in Learning to Evolve with Convergence Guarantee via Neural Unrolling. Yaohua Liu and Kay Chen Tan show that unshared parameters across steps allow an optimizer to develop these stage-specific behaviors. Similarly, in Mean-Field Networks, Richard Zemel and Yujia Li found that untying weights across layers allowed lower layers to capture broad features while higher layers focused on fine-grained refinements.
Automating the Hyperparameter Search
Perhaps the most practical benefit of this research is the elimination of manual tuning. In complex tasks like transductive few-shot learning, finding the right balance of entropy and class balance is notoriously difficult.
By unrolling the Expectation-Maximization (EM) algorithm, Fereshteh Shakeri and Ismail Ben Ayed created UNEM. Instead of a researcher spending days grid-searching for the best static hyperparameters, the model learns a "parameter trajectory." This trajectory adapts to the specific dataset, leading to accuracy gains of up to 10% over traditional iterative methods that are stuck with a single, "one-size-fits-all" parameter.