Single-Timescale Actor-Critic Methods

This direction is split into other directions: simultaneous actor-critic optimization. Its last version stays readable and citable here.

Version 1 of 195 papers137 conceptsfrozen 2026-10-04

In reinforcement learning, the Actor-Critic (AC) framework is the standard workhorse for continuous control. Theoretically, these algorithms were long described as "two-timescale" or "double-loop" processes: the Critic (the value function) must converge to an accurate estimate of the current policy before the Actor (the policy) is allowed to take a step. If the Actor moves too quickly based on a "noisy" or inaccurate Critic, the system risks divergence.

However, modern deep RL practitioners rarely follow this prescription. In practice, the Actor and Critic are updated simultaneously using the same stream of data and comparable learning rates. This discrepancy—where practice ignores the "safety" of nested loops—has led to a surge of research proving that single-timescale actor-critic methods are not just a heuristic hack, but are theoretically optimal and robust.

The Efficiency of Simultaneous Updates

The core intuition behind single-timescale methods is that the Actor does not need a perfect value estimate to improve; it only needs a "good enough" gradient direction. By updating both components concurrently, the algorithm avoids the massive computational overhead of waiting for the Critic to reach full convergence at every step.

Zuyue Fu, Zhuoran Yang, and Zhaoran Wang demonstrated in Single-Timescale Actor-Critic Provably Finds Globally Optimal Policy that simultaneous updates can achieve global optimality. This work bridged the gap between the "black box" of deep RL and rigorous optimization theory, showing that the coupling between the evolving policy and the evolving value function does not inherently prevent convergence.

Taming the Coupling with Finite-Time Analysis

The primary challenge in analyzing single-timescale methods is the "tight coupling" of the components. Because the Actor and Critic change at the same time, the Critic is trying to estimate a "moving target" (the value of a changing policy), while the Actor is following a "shifting gradient" (the estimates of a changing Critic).

Recent work has focused on providing "finite-time" or "non-asymptotic" bounds to prove how fast these systems converge:

Robustness Across Diverse Domains

The single-timescale paradigm has proven surprisingly versatile, maintaining stability even when the underlying problem becomes more complex.

Mean-Field and Multi-Agent Games

In large-scale systems like Mean-Field Games (MFGs), where an agent interacts with a vast population, the "learning while playing" approach allows the agent’s policy and the population’s aggregate state to evolve simultaneously. Qiaomin Xie and colleagues showed in Learning While Playing in Mean-Field Games: Convergence and Optimality that this single-loop scheme reduces computational overhead without sacrificing Nash Equilibrium guarantees. Similarly, Hongyi Guo applied this to decentralized zero-sum games in Decentralized Single-Timescale Actor-Critic on Zero-Sum Two-Player Stochastic Games.

Evolving Environments and Rewards

In many practical scenarios, such as curriculum learning or reward shaping, the reward function itself changes over time. Yu Chen and Longbo Huang proved in Finite-time Convergence Analysis of Actor-Critic with Evolving Reward that single-timescale AC methods still converge at the optimal $\mathcal{O}(1/\sqrt{T})$ rate, provided the rewards evolve slowly enough.

Federated and Distributed Learning

To handle data privacy and scale, Xiaowen Gong introduced Single-Loop Federated Actor-Critic, demonstrating that multiple agents can update their local actors and critics simultaneously while achieving a linear speedup in convergence as more agents join the network.

Beyond Simple Gradients: Momentum and Regularization

To reach the theoretical "speed limit" of reinforcement learning, researchers are now combining single-timescale updates with advanced optimization techniques. Navdeep Kumar and Tehila Dahan showed in Optimal Sample Complexity for Single Time-Scale Actor-Critic with Momentum that by incorporating stochastic recursive momentum (STORM) into the critic, a single-timescale algorithm can finally achieve the optimal $\mathcal{O}(\epsilon^{-2})$ complexity for global optimality.

Furthermore, Han-Dong Lim and Donghwan Lee demonstrated in Regularized Q-Learning that single-timescale updates, when properly regularized, can even overcome the "deadly triad" of off-policy learning, function approximation, and bootstrapping—a challenge that has historically caused RL algorithms to diverge.

By treating the Actor and Critic as a single, coupled dynamical system rather than two separate optimization problems, this research direction has successfully aligned RL theory with the high-performance simultaneous update strategies used in the real world.

Papers in this version

Showing 60 of 95 papers.

History & citationVersion 1 · 2026-10-04

Split into other directions · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_single_timescale_actor_critic_methods_80a78c45,
  title = {Single-Timescale Actor-Critic Methods},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/single-timescale-actor-critic-methods/rdr_80a78c45a5234fdeba9eec51256864b8},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_single_timescale_actor_critic_methods_80a78c45_v1,
  title = {Single-Timescale Actor-Critic Methods},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/single-timescale-actor-critic-methods/rdr_80a78c45a5234fdeba9eec51256864b8/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}