In reinforcement learning, actor-critic algorithms split the burden of finding an optimal policy into two distinct roles: the actor, which learns the control policy, and the critic, which evaluates the quality of that policy by estimating its value function.
Historically, optimization theory dictated a strict separation between these two components. To guarantee convergence, researchers relied on a two-timescale assumption or a double-loop architecture. In these traditional frameworks, the critic is required to either fully converge or update at an infinitesimally faster rate than the actor. The intuition was straightforward: if the actor updates its policy based on an inaccurate, noisy critic, the errors will compound, leading to instability or catastrophic divergence.
However, a major gap persisted between theory and practice. Modern deep reinforcement learning practitioners routinely ignore this separation, updating both the actor and the critic simultaneously within the same training iteration using comparable learning rates.
A growing body of research has successfully closed this gap by proving that single-timescale simultaneous optimization is not only stable but theoretically optimal.
The Intuition: Why Simultaneous Updates Work
If the critic's value estimates are always imperfect because it never fully converges before the policy changes, why doesn't the system explode? The answer lies in shifting our perspective from static convergence to dynamic tracking.
1. Incremental Tracking Over Full Convergence
The critic does not need to find the absolute true value function of the current policy at every step. It merely needs to track the value function as the policy evolves. As noted by Yue Wu et al., simultaneous updates allow the critic to incrementally track the value function across iterations, preserving learned information rather than discarding progress between decoupled training phases.
2. Bounding the Coupling Error
Mathematically, the actor and critic form a coupled dynamical system. Instead of analyzing them in isolation, recent work treats their co-evolution as a single unified process. Huaqing Xiong et al. demonstrated that by using constant stepsizes, the coupling error between the actor and the critic can be strictly bounded. This bound ensures that the errors introduced by an imperfect critic do not grow uncontrollably, rendering the traditional requirement for a faster critic unnecessary.
3. The Efficiency of Frequent Small Steps
Waiting for a critic to fully converge wastes massive amounts of data on an intermediate policy that will soon be discarded. Jingliang Duan et al. showed that frequent policy improvements based on imperfect, single-sample value estimates actually accelerate the search for an optimal policy. Similarly, Yudan Wang et al. proved that single-loop architectures can continuously learn from a single continuous trajectory, maximizing data throughput and eliminating the computational bottlenecks of nested loops.
Traditional Double-Loop:
[Critic Updates to Convergence] -> [Actor Takes One Step] -> [Repeat Loop] (Data Inefficient)
Single-Timescale Simultaneous Loop:
[Actor Step + Critic Step] ───(Continuous Stream of Data)───> [Global Optimum] (Data Efficient)
Theoretical Foundations and Convergence Rates
By treating simultaneous updates as a single-loop stochastic optimization problem, recent papers have established rigorous finite-time convergence guarantees that match or exceed their multi-timescale counterparts.
- Global Optimality: Zuyue Fu et al. provided some of the first global optimality guarantees for single-timescale actor-critic algorithms under both linear and deep neural network approximations, establishing a sublinear convergence rate of $O(K^{-1/2})$.
- Sample Complexity: For continuous state-action spaces, Jingliang Duan et al. and Xuyang Chen et al. proved that single-timescale updates under realistic sequential (Markovian) sampling achieve a sample complexity of $\mathcal{O}(\epsilon^{-2})$ to reach an $\epsilon$-optimal solution. This matches the theoretical efficiency of standard Stochastic Gradient Descent (SGD).
- Deep Networks: Extending these proofs to practical deep learning architectures, Haoxing Tian et al. showed that simultaneous updates of actor and critic networks of arbitrary depth stably converge to a stationary point at a rate of $\tilde{O}(T^{-0.5})$.
Broadening the Paradigm
The success of single-timescale optimization has led to its adoption across several complex frontiers in reinforcement learning, proving that simultaneous updates can stabilize systems previously thought to require complex scheduling.
Multi-Agent Systems and Games
In competitive environments like zero-sum stochastic games, finding a Nash equilibrium traditionally requires solving nested sub-games at every step. Hongyi Guo et al. and Yuepeng Yang et al. proved that updating policies and value estimates concurrently on a single timescale efficiently converges to equilibrium when stabilized by policy regularization, bypassing the need for nested optimization loops.
Federated Learning
In distributed settings where multiple agents learn a shared policy across non-identical environments, synchronization costs can be crippling. Ye Zhu et al. introduced a single-loop federated actor-critic architecture that updates both components simultaneously across a network of agents, maintaining stability and high sample efficiency without requiring full critic convergence before aggregation.
Taming the Off-Policy "Deadly Triad"
Off-policy learning combined with function approximation and bootstrapping is notoriously unstable—a phenomenon known as the deadly triad. To combat this, traditional methods introduce complex dual-learning rates or multi-timescale schedules. However, Zaiwei Chen et al. and Han-Dong Lim et al. demonstrated that by leveraging structural regularizations or multi-step returns, stable off-policy learning can be achieved using a single-timescale update rule, drastically simplifying hyperparameter tuning while maintaining robust convergence.
Key Foundational Work
Temporal Difference Based Actor Critic Learning - Convergence and Neural Implementation (2008) — contributed 2 concept elements to this direction.
Least squares temporal difference actor-critic methods with applications to robot motion control (2011) — contributed 1 concept elements to this direction.