In modern machine learning and signal processing, we rarely encounter optimization problems that are perfectly smooth. Instead, we often face "composite" objectives: functions that are the sum of a smooth, data-driven loss (like least squares) and a non-smooth regularizer (like the $L_1$ norm for sparsity).
Standard optimization tools face a dilemma here. Gradient descent requires derivatives, which don't exist at the "kinks" of non-smooth functions. Conversely, general non-smooth methods like subgradient descent are notoriously slow, converging at a rate of $O(1/\sqrt{t})$. The research direction of composite optimization works by breaking this dilemma: rather than treating the objective as a single "black box," it exploits the individual structure of each component to achieve faster, often optimal, convergence rates.
The Power of Splitting
The core intuition behind this field is operator splitting. If an objective is composed of a smooth part $f(x)$ and a non-smooth part $g(x)$, we should treat them differently. Hu et al. argue that by separating these components, we can apply Nesterov-style acceleration to the smooth part while handling the non-smooth part through a "proximal step"—a specialized operation that solves a simplified version of the non-smooth problem.
This decomposition allows algorithms to reach the theoretical lower bounds for convergence. As Doikov establishes, understanding the interplay between these smooth and non-smooth parts is essential for determining the fundamental limits of how fast any algorithm can possibly solve these problems.
Accelerating Through Structure
Why does this structural awareness lead to better performance? In many cases, the non-smooth part of the function is "simple" or "proximable," meaning we can calculate its effect exactly even if we can't take its derivative.
- Smoothing and Acceleration: Gray et al. demonstrate that we can accelerate stochastic gradient descent for "kinky" loss functions by intelligently smoothing the non-smooth parts over time. This allows the algorithm to behave like a smooth optimizer while still respecting the original non-smooth objective, achieving an optimal $O(1/t)$ rate for strongly convex problems.
- Proximal Mappings: Algorithms like FISTA, analyzed by Kim et al., use the proximal gradient method to minimize composite functions. By applying a gradient step to the smooth part and then a "shrinkage" or "thresholding" step to the non-smooth part, these methods maintain the $O(1/t^2)$ convergence rate of accelerated gradient descent despite the lack of global differentiability.
- Identifying Activity: A fascinating property of these methods is their ability to identify the "active" constraints or non-smooth features of a problem. Liang et al. show that Forward-Backward type methods can identify these active components early in the process, after which the algorithm begins to converge linearly, explaining why these methods often perform better in practice than their worst-case theory suggests.
Scaling to Complex Systems
As problems grow in scale and complexity, the composite framework adapts by using more sophisticated "oracles" for each component.
- Higher-Order Methods: For high-dimensional problems where first-order methods might be too slow, Doikov et al. propose using cubic regularization for the smooth parts of the objective while keeping exact models for the non-smooth parts. This hybrid approach provides the stability of second-order methods without the prohibitive cost of full-space updates.
- Distributed and Manifold Optimization: The composite structure is also vital for decentralized systems. Wang et al. developed distributed ADMM variants that allow multiple agents to reach a consensus on a composite objective, while Wang et al. extended these ideas to non-Euclidean spaces like the Stiefel manifold.
- Heterogeneous Components: Sometimes, an objective is a sum of three or more parts. Tang et al. introduced three-operator splitting to handle cases where an objective includes a smooth term, a non-smooth term, and a third term that might be a complex prior (like a deep-learning-based denoiser).
A Compositional Perspective
Ultimately, this research direction views optimization not as a monolithic task, but as a system of interacting parts. Stein et al. even suggest a categorical framework where convex optimization problems are treated as "bifunctions" that can be composed like building blocks. By recognizing that different parts of a problem require different mathematical tools—gradients for the smooth, proximal steps for the non-smooth, and cubic models for the complex—researchers have created a toolkit that is both mathematically rigorous and practically efficient for the most challenging problems in modern AI.
Key Foundational Work
Accelerated Gradient Methods for Stochastic Optimization and Online Learning (2009) — contributed 2 concept elements to this direction.
Stochastic Smoothing for Nonsmooth Minimizations: Accelerating SGD by Exploiting Structure (2012) — contributed 1 concept elements to this direction.