Aligning Critic Gradients with Policy Improvement

This direction is split into other directions: regularization-induced bias. Its last version stays readable and citable here.

Version 1 of 146 papers50 conceptsfrozen 2026-10-04

In the standard Actor-Critic architecture, we divide the labor of learning into two roles: the Actor, which learns a policy to maximize rewards, and the Critic, which estimates the value of states to guide the Actor. While this division is intuitive, a growing body of research suggests that these two components are often working at cross-purposes.

The fundamental issue is an objective mismatch. The Critic is typically trained to be a good "bookkeeper"—minimizing the error between its predictions and observed rewards. However, the Actor doesn't need a bookkeeper; it needs a "coach" that provides accurate directional gradients for improvement.

The Bookkeeper vs. The Coach

Most reinforcement learning (RL) algorithms train the Critic using Mean Squared Error (MSE) or Temporal Difference (TD) learning. As Sharan Vaswani and colleagues argue in Decision-Aware Actor-Critic with Function Approximation and Theoretical Guarantees, this creates a disconnect. A Critic can be highly accurate at predicting the total reward of a state (low MSE) while being completely wrong about which action is slightly better than another.

This discrepancy is most visible in continuous control. To update the policy, the Actor looks at the "slope" (gradient) of the Critic’s value surface. If the Critic is trained only to match the magnitude of rewards, its surface might be "bumpy" or noisy in ways that lead the Actor astray. Baturay Sağlam and Dionysis Kalogerias address this in Compatible Gradient Approximations for Actor-Critic Algorithms, noting that even precise value predictions can produce inaccurate gradients. To fix this, researchers like Fabian Schramm propose First-order Sobolev Reinforcement Learning, which forces the Critic to match not just the target values, but also the derivatives of the environment dynamics.

The Regularization Trap

To prevent the Actor from collapsing into a single sub-optimal action too early, we often use Entropy Regularization. This rewards the agent for being "surprised" or staying stochastic. While this helps exploration, it introduces a persistent bias.

As Lior Shani and Shie Mannor point out in their work on Adaptive Trust Region Policy Optimization, adding an entropy bonus fundamentally changes the goal of the agent. The agent is no longer solving the original problem; it is solving a "smoothed" version where randomness is inherently valuable.

This leads to asymptotic inconsistency. In Monte Carlo Tree Search with Boltzmann Exploration, Michael Painter and Bruno Lacerda demonstrate that if the entropy bonus is fixed, the agent may never converge to the true optimal policy. It remains in a state of "perpetual exploration," choosing sub-optimal actions because the reward for randomness outweighs the reward for being correct. The solution, explored by David Siska and others, is entropy annealing—gradually reducing the regularization to zero so the agent eventually focuses on the true objective.

Synchronizing the Two Players

If the Actor and Critic have different goals, how do we force them to cooperate? One powerful approach is to derive both components from the same mathematical foundation.

In Boosting the Actor with Dual Critic, Albert Eaton Shaw and Bo Dai frame RL as a Lagrangian dual problem. Instead of the Critic minimizing TD error and the Actor maximizing reward, they treat the interaction as a two-player cooperative game where both optimize the same objective. This ensures that every update the Critic makes is mathematically guaranteed to support the Actor’s progress.

Similarly, the VISA-VIS method by Daniel Lee and Bart Selman addresses "internal contradictions" in superhuman agents like AlphaZero. By forcing the policy network (the move selector) and the value network (the board evaluator) into alignment, they ensure the agent doesn't suggest a move that its own evaluation network thinks is a losing one.

Why It Works

The shift toward "decision-aware" or "aligned" actor-critic methods works because it acknowledges that representation matters more than approximation. In high-dimensional spaces, a Critic cannot be perfect everywhere. By aligning the Critic's objective with the Actor's needs, we ensure that the Critic is "most accurate" in the specific ways that help the Actor improve—namely, in the direction and magnitude of the policy gradient.

When the "bookkeeper" becomes a "coach," the learning process stabilizes, requires less data, and avoids the biased pitfalls of simple entropy-based exploration.

Papers in this version

History & citationVersion 1 · 2026-10-04

Split into other directions · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_aligning_critic_gradients_with_policy_im_e40e54ff,
  title = {Aligning Critic Gradients with Policy Improvement},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/aligning-critic-gradients-with-policy-improvement/rdr_e40e54ffb4284fde94ade90921aa3e05},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_aligning_critic_gradients_with_policy_im_e40e54ff_v1,
  title = {Aligning Critic Gradients with Policy Improvement},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/aligning-critic-gradients-with-policy-improvement/rdr_e40e54ffb4284fde94ade90921aa3e05/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}