Single Timescale Actor Critic Optimization

49 concepts36 papers1 proposal

In reinforcement learning, actor-critic algorithms split the burden of finding an optimal policy into two distinct roles: the actor, which learns the control policy, and the critic, which evaluates the quality of that policy by estimating its value function.

Historically, optimization theory dictated a strict separation between these two components. To guarantee convergence, researchers relied on a two-timescale assumption or a double-loop architecture. In these traditional frameworks, the critic is required to either fully converge or update at an infinitesimally faster rate than the actor. The intuition was straightforward: if the actor updates its policy based on an inaccurate, noisy critic, the errors will compound, leading to instability or catastrophic divergence.

However, a major gap persisted between theory and practice. Modern deep reinforcement learning practitioners routinely ignore this separation, updating both the actor and the critic simultaneously within the same training iteration using comparable learning rates.

A growing body of research has successfully closed this gap by proving that single-timescale simultaneous optimization is not only stable but theoretically optimal.

The Intuition: Why Simultaneous Updates Work

If the critic's value estimates are always imperfect because it never fully converges before the policy changes, why doesn't the system explode? The answer lies in shifting our perspective from static convergence to dynamic tracking.

1. Incremental Tracking Over Full Convergence

The critic does not need to find the absolute true value function of the current policy at every step. It merely needs to track the value function as the policy evolves. As noted by Yue Wu et al., simultaneous updates allow the critic to incrementally track the value function across iterations, preserving learned information rather than discarding progress between decoupled training phases.

2. Bounding the Coupling Error

Mathematically, the actor and critic form a coupled dynamical system. Instead of analyzing them in isolation, recent work treats their co-evolution as a single unified process. Huaqing Xiong et al. demonstrated that by using constant stepsizes, the coupling error between the actor and the critic can be strictly bounded. This bound ensures that the errors introduced by an imperfect critic do not grow uncontrollably, rendering the traditional requirement for a faster critic unnecessary.

3. The Efficiency of Frequent Small Steps

Waiting for a critic to fully converge wastes massive amounts of data on an intermediate policy that will soon be discarded. Jingliang Duan et al. showed that frequent policy improvements based on imperfect, single-sample value estimates actually accelerate the search for an optimal policy. Similarly, Yudan Wang et al. proved that single-loop architectures can continuously learn from a single continuous trajectory, maximizing data throughput and eliminating the computational bottlenecks of nested loops.

Traditional Double-Loop:
[Critic Updates to Convergence] -> [Actor Takes One Step] -> [Repeat Loop] (Data Inefficient)

Single-Timescale Simultaneous Loop:
[Actor Step + Critic Step] ───(Continuous Stream of Data)───> [Global Optimum] (Data Efficient)

Theoretical Foundations and Convergence Rates

By treating simultaneous updates as a single-loop stochastic optimization problem, recent papers have established rigorous finite-time convergence guarantees that match or exceed their multi-timescale counterparts.

  • Global Optimality: Zuyue Fu et al. provided some of the first global optimality guarantees for single-timescale actor-critic algorithms under both linear and deep neural network approximations, establishing a sublinear convergence rate of $O(K^{-1/2})$.
  • Sample Complexity: For continuous state-action spaces, Jingliang Duan et al. and Xuyang Chen et al. proved that single-timescale updates under realistic sequential (Markovian) sampling achieve a sample complexity of $\mathcal{O}(\epsilon^{-2})$ to reach an $\epsilon$-optimal solution. This matches the theoretical efficiency of standard Stochastic Gradient Descent (SGD).
  • Deep Networks: Extending these proofs to practical deep learning architectures, Haoxing Tian et al. showed that simultaneous updates of actor and critic networks of arbitrary depth stably converge to a stationary point at a rate of $\tilde{O}(T^{-0.5})$.

Broadening the Paradigm

The success of single-timescale optimization has led to its adoption across several complex frontiers in reinforcement learning, proving that simultaneous updates can stabilize systems previously thought to require complex scheduling.

Multi-Agent Systems and Games

In competitive environments like zero-sum stochastic games, finding a Nash equilibrium traditionally requires solving nested sub-games at every step. Hongyi Guo et al. and Yuepeng Yang et al. proved that updating policies and value estimates concurrently on a single timescale efficiently converges to equilibrium when stabilized by policy regularization, bypassing the need for nested optimization loops.

Federated Learning

In distributed settings where multiple agents learn a shared policy across non-identical environments, synchronization costs can be crippling. Ye Zhu et al. introduced a single-loop federated actor-critic architecture that updates both components simultaneously across a network of agents, maintaining stability and high sample efficiency without requiring full critic convergence before aggregation.

Taming the Off-Policy "Deadly Triad"

Off-policy learning combined with function approximation and bootstrapping is notoriously unstable—a phenomenon known as the deadly triad. To combat this, traditional methods introduce complex dual-learning rates or multi-timescale schedules. However, Zaiwei Chen et al. and Han-Dong Lim et al. demonstrated that by leveraging structural regularizations or multi-step returns, stable off-policy learning can be achieved using a single-timescale update rule, drastically simplifying hyperparameter tuning while maintaining robust convergence.

Key Foundational Work

Temporal Difference Based Actor Critic Learning - Convergence and Neural Implementation (2008) — contributed 2 concept elements to this direction.
Least squares temporal difference actor-critic methods with applications to robot motion control (2011) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_simultaneous_actor_critic_optimization_1db291d6,
  title = {simultaneous actor-critic optimization},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/simultaneous-actor-critic-optimization/rdr_1db291d6fc934eaa967e4aa62160a9b5},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_simultaneous_actor_critic_optimization_1db291d6_v1,
  title = {simultaneous actor-critic optimization},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/simultaneous-actor-critic-optimization/rdr_1db291d6fc934eaa967e4aa62160a9b5/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake