In reinforcement learning, the actor-critic framework splits the agent's brain into two components: an actor that decides how to behave (the policy) and a critic that evaluates how good those decisions are (the value function). While this division of labor is highly effective for continuous control and complex reasoning tasks, it introduces a fundamental mathematical catch-22.
The actor relies on the critic's value estimates to compute policy gradients and improve its behavior. Simultaneously, the critic relies on data generated by the actor's current policy to evaluate state-action values. This circular dependency creates an interleaved optimization challenge known as the moving target problem. Because both networks are updated concurrently, each component optimizes against a shifting landscape driven by the other, frequently leading to training instability, high hyperparameter sensitivity, or catastrophic divergence.
The Destructive Feedback Loop of Value and Policy Churn
To understand why this coupling fails, we must look at how errors propagate through the system. If the critic contains even minor estimation errors, it provides inaccurate gradients to the actor. The actor then updates its policy based on these flawed signals, shifting the agent's data distribution. This new distribution introduces unfamiliar states and actions into the training pipeline, which further degrades the critic’s ability to make accurate predictions.
Hongyao Tang et al. formalize this phenomenon as a "chain effect" of value and policy churn. Unintended changes in value predictions trigger erratic policy shifts, which reciprocally amplify future value estimation errors. As Simone Parisi et al. point out, traditional stabilization methods often treat the actor and the critic in isolation—such as bounding policy updates or smoothing value targets—while completely neglecting the instability arising from their direct interaction. This unmanaged feedback loop can cause training to collapse entirely, an issue that also hinders multi-step reasoning performance when fine-tuning large language models via reinforcement learning, as observed in the development of DAPO.
Theoretical Boundaries of Single-Timescale Updates
Historically, the primary theoretical tool used to guarantee actor-critic stability was the two-timescale framework. By forcing the actor to learn at a significantly slower rate than the critic, the critic is given enough time to converge to the true value of the current policy before the policy shifts again.
In practice, however, modern deep reinforcement learning algorithms utilize single-timescale updates, where both networks are updated simultaneously using identical or comparable learning rates. This creates a highly coupled dynamical system that is notoriously difficult to analyze.
Recent theoretical breakthroughs have begun to map out why and when these simultaneous updates actually converge:
- Linear Quadratic Regulators (LQR): Zhuoran Yang et al. provided the first nonasymptotic global convergence guarantees for actor-critic algorithms in continuous control, proving that they can find globally optimal policies at a linear rate despite the moving target effect. Xuyang Chen et al. and Jingliang Duan et al. extended this to show that single-timescale updates under sequential, Markovian sampling can achieve global optimality with a sample complexity that matches standard stochastic gradient descent.
- Deep Neural Networks: Moving beyond linear approximations, Haoxing Tian et al. proved that actor-critic methods using multi-layer neural networks of arbitrary depth can successfully converge to stationary points. Furthermore, Yufeng Zhang et al. demonstrated that this joint optimization allows the networks to escape the restrictive Neural Tangent Kernel (NTK) regime, meaning the actor and critic actively collaborate to learn meaningful, data-dependent feature representations during training.
Algorithmic Solutions for Decoupling and Regularization
To mitigate the moving target problem without sacrificing sample efficiency, modern algorithms introduce explicit mechanisms to break or dampen the actor-critic feedback loop.
1. Error-Aware Policy Regularization
One intuitive fix is to stop the actor from moving when the critic is guessing blindly. Simone Parisi et al. introduced a TD-regularized framework that penalizes the actor's objective function whenever the critic's temporal difference (TD) error is high. By anchoring policy steps to the reliability of the value signal, the agent avoids taking large, destructive steps based on uncalibrated value estimates.
2. Target Modification and Memory Stacks
Instead of forcing the critic to constantly chase a moving policy, algorithms can change the optimization target itself. Kevin Tan et al. developed an approach that forces the critic to target the optimal action-value function ($Q^*$) rather than the value of the volatile current policy, combining this with "rare-switching" updates to isolate the two components. From a memory perspective, Alena Shilova et al. introduced a method that maintains a finite stack of historical Q-functions to stabilize Policy Mirror Descent, smoothing out sudden oscillations in the value landscape.
3. State-Occupancy Matching
In offline and goal-conditioned reinforcement learning, where the lack of online exploration makes error propagation particularly severe, interleaved optimization can be bypassed entirely. Algorithms like GoFAR, developed by Yecheng Jason Ma et al., and its theoretical extensions by Hanlin Zhu et al., reformulate reinforcement learning as a state-occupancy matching problem. This allows the value function and policy to be solved via a stable, two-step regression framework, completely avoiding the computational instability of minimax or interleaved optimization.
Bypassing the Loop via Actor-Free Architectures
The most definitive way to solve the actor-critic coupling problem is to eliminate one of the moving targets entirely. In discrete action spaces, value-based methods like Deep Q-Networks (DQN) are highly stable because finding the optimal action simply requires computing a maximum over a finite set of outputs:
$$\arg\max_{a} Q(s, a)$$
In continuous action spaces (such as robotics), maximizing an arbitrary neural network over an infinite range of actions is mathematically difficult, which is why the field originally gravitated toward dual-network actor-critic architectures.
To eliminate the actor network while retaining continuous control capabilities, recent work focuses on designing structurally maximizable value functions.
Actor-Critic Paradigm (Two Moving Targets):
[Actor Policy] ───Updates based on───> [Critic Value]
▲ │
└─────────Shifts targets for─────────────┘
Actor-Free Paradigm (Single Unified Optimization):
[Structurally Maximizable Q-Function] ───> Direct Convex Optimization ───> Action
Yigit Korkmaz et al. developed continuous control networks that maintain analytical maximizability, allowing agents to perform direct value-based control without an actor. Similarly, Ruoqi Zhang et al. leveraged Input-Convex Neural Networks (ICNNs) to guarantee that the value landscape remains convex with respect to the action space. This allows the optimal action to be found efficiently at every step via standard convex optimization.
By bringing continuous-time control theory into deep learning, Jeongho Kim 0002 et al. derived a new class of Hamilton-Jacobi-Bellman equations specifically tailored for Q-functions. This mathematical restructuring collapses the dual-network problem into a single, unified value-function optimization, completely neutralizing the reciprocal churn that has long destabilized actor-critic learning.
Key Foundational Work
TD-Regularized Actor-Critic Methods (2018) — contributed 2 concept elements to this direction.
Provably Global Convergence of Actor-Critic: A Case for Linear Quadratic Regulator with Ergodic Cost (2019) — contributed 1 concept elements to this direction.