In reinforcement learning, the Actor-Critic (AC) framework is the standard workhorse for continuous control. Theoretically, these algorithms were long described as "two-timescale" or "double-loop" processes: the Critic (the value function) must converge to an accurate estimate of the current policy before the Actor (the policy) is allowed to take a step. If the Actor moves too quickly based on a "noisy" or inaccurate Critic, the system risks divergence.
However, modern deep RL practitioners rarely follow this prescription. In practice, the Actor and Critic are updated simultaneously using the same stream of data and comparable learning rates. This discrepancy—where practice ignores the "safety" of nested loops—has led to a surge of research proving that single-timescale actor-critic methods are not just a heuristic hack, but are theoretically optimal and robust.
The Efficiency of Simultaneous Updates
The core intuition behind single-timescale methods is that the Actor does not need a perfect value estimate to improve; it only needs a "good enough" gradient direction. By updating both components concurrently, the algorithm avoids the massive computational overhead of waiting for the Critic to reach full convergence at every step.
Zuyue Fu, Zhuoran Yang, and Zhaoran Wang demonstrated in Single-Timescale Actor-Critic Provably Finds Globally Optimal Policy that simultaneous updates can achieve global optimality. This work bridged the gap between the "black box" of deep RL and rigorous optimization theory, showing that the coupling between the evolving policy and the evolving value function does not inherently prevent convergence.
Taming the Coupling with Finite-Time Analysis
The primary challenge in analyzing single-timescale methods is the "tight coupling" of the components. Because the Actor and Critic change at the same time, the Critic is trying to estimate a "moving target" (the value of a changing policy), while the Actor is following a "shifting gradient" (the estimates of a changing Critic).
Recent work has focused on providing "finite-time" or "non-asymptotic" bounds to prove how fast these systems converge:
- Sample Complexity: Xuyang Chen and Lin Zhao proved in Finite-Time Analysis of Single-Timescale Actor-Critic that these algorithms reach a stationary point with a sample complexity of $\widetilde{\mathcal{O}}(\epsilon^{-2})$. This matches the efficiency of standard Stochastic Gradient Descent, suggesting that the complexity of the RL setting doesn't necessarily slow down the underlying optimization.
- Continuous Spaces: In On the Convergence of Continuous Single-timescale Actor-critic, Xuyang Chen and Lin Zhao extended these guarantees to continuous state-action spaces, which are critical for robotics and control.
- Tightening the Bounds: Navdeep Kumar and Priyank Agrawal introduced an "ODE domination" framework in On the Convergence of Single-Timescale Actor-Critic, improving the proven sample complexity for global optimality from $\mathcal{O}(\epsilon^{-4})$ to $\mathcal{O}(\epsilon^{-3})$.
Robustness Across Diverse Domains
The single-timescale paradigm has proven surprisingly versatile, maintaining stability even when the underlying problem becomes more complex.
Mean-Field and Multi-Agent Games
In large-scale systems like Mean-Field Games (MFGs), where an agent interacts with a vast population, the "learning while playing" approach allows the agent’s policy and the population’s aggregate state to evolve simultaneously. Qiaomin Xie and colleagues showed in Learning While Playing in Mean-Field Games: Convergence and Optimality that this single-loop scheme reduces computational overhead without sacrificing Nash Equilibrium guarantees. Similarly, Hongyi Guo applied this to decentralized zero-sum games in Decentralized Single-Timescale Actor-Critic on Zero-Sum Two-Player Stochastic Games.
Evolving Environments and Rewards
In many practical scenarios, such as curriculum learning or reward shaping, the reward function itself changes over time. Yu Chen and Longbo Huang proved in Finite-time Convergence Analysis of Actor-Critic with Evolving Reward that single-timescale AC methods still converge at the optimal $\mathcal{O}(1/\sqrt{T})$ rate, provided the rewards evolve slowly enough.
Federated and Distributed Learning
To handle data privacy and scale, Xiaowen Gong introduced Single-Loop Federated Actor-Critic, demonstrating that multiple agents can update their local actors and critics simultaneously while achieving a linear speedup in convergence as more agents join the network.
Beyond Simple Gradients: Momentum and Regularization
To reach the theoretical "speed limit" of reinforcement learning, researchers are now combining single-timescale updates with advanced optimization techniques. Navdeep Kumar and Tehila Dahan showed in Optimal Sample Complexity for Single Time-Scale Actor-Critic with Momentum that by incorporating stochastic recursive momentum (STORM) into the critic, a single-timescale algorithm can finally achieve the optimal $\mathcal{O}(\epsilon^{-2})$ complexity for global optimality.
Furthermore, Han-Dong Lim and Donghwan Lee demonstrated in Regularized Q-Learning that single-timescale updates, when properly regularized, can even overcome the "deadly triad" of off-policy learning, function approximation, and bootstrapping—a challenge that has historically caused RL algorithms to diverge.
By treating the Actor and Critic as a single, coupled dynamical system rather than two separate optimization problems, this research direction has successfully aligned RL theory with the high-performance simultaneous update strategies used in the real world.