Compatible Function Approximation for Natural Policy Gradients

This direction is split into other directions: compatible function approximation. Its last version stays readable and citable here.

Version 1 of 140 papers55 conceptsfrozen 2026-10-04

In reinforcement learning, we often treat the "Actor" (the policy) and the "Critic" (the value function) as two independent modules. The Actor tries to improve its behavior, while the Critic tries to predict the rewards that behavior will generate. However, a long line of research suggests that these two components are not just partners—they are mathematically two sides of the same coin.

The core idea is Compatible Function Approximation. It posits that for an Actor-Critic system to be truly efficient, the architecture of the Critic must be dictated by the structure of the Actor. When these two are "compatible," the Critic stops being a mere predictor and becomes a direct map of the policy's steepest ascent direction.

The Problem of Parameter Sensitivity

Standard policy gradient methods are notoriously sensitive to how a policy is parameterized. If you describe a robot arm's joint in degrees versus radians, the "steepest" gradient direction in parameter space changes, even though the physical movement is identical. This is because standard gradients move in the space of parameters ($\theta$), not the space of behaviors (the probability distribution $\pi$).

To fix this, Sham M. Kakade introduced the Natural Policy Gradient (NPG). Instead of following the raw gradient, NPG follows the "natural" gradient, which is invariant to re-parameterization. It looks at the underlying geometry of the probability distributions using the Fisher Information Matrix. While mathematically elegant, NPG originally seemed computationally daunting because it required calculating and inverting a massive matrix at every step.

The Compatibility Insight

The breakthrough came from realizing that we don't actually need to compute the Fisher Information Matrix explicitly. Vijay R. Konda and John N. Tsitsiklis showed in Actor-Critic Algorithms that if the Critic's features are chosen to be the "score function" of the policy—specifically the gradient of the log-probability $\nabla_\theta \log \pi_\theta(s, a)$—the Critic becomes "compatible."

Why does this work? Intuition suggests that the Critic should only care about the parts of the value function that the Actor can actually influence. By using the policy's own log-gradients as the basis for the Critic, we ensure the Critic captures exactly the information necessary to update the policy. As Yudan Wang and colleagues note in recent theoretical work, this alignment eliminates the persistent approximation error that usually plagues linear value estimation.

The Critic as the Gradient

When the Critic is compatible, a remarkable mathematical equivalence emerges: the weights of the Critic are the Natural Policy Gradient.

In Applying the Episodic Natural Actor-Critic Architecture to Motor Primitive Learning, Jan Peters demonstrated that by simply performing a linear regression to fit the Critic's weights, you are simultaneously computing the natural gradient. This bypasses the need for matrix inversion entirely. This efficiency allowed robots to learn complex motor skills, like a baseball swing, in just a few hundred trials—a massive speedup over standard methods.

This equivalence has been extended to various domains:

  • Geometry: J. Andrew Bagnell and Jeff Schneider proved in Covariant Policy Search that this approach is rigorously "covariant," meaning the learning trajectory remains the same regardless of how you label your parameters.
  • Kernels: Guy Lever and Ronnie Stafford extended this to Reproducing Kernel Hilbert Spaces, allowing policies to adapt their complexity without manual feature engineering.
  • Second-Order Updates: Arash Bahari Kordabad and others have shown that specific quadratic structures in the Critic can even provide curvature (Hessian) information for Quasi-Newton updates.

Modern Perspectives: Alignment and Consistency

Today, the principle of compatibility is evolving into a broader concept of "gradient alignment."

In deep RL, we often struggle with "gradient conflict," where the Actor and Critic pull the network parameters in different directions. Changnan Xiao and the authors of CASA argue for forcing the gradients of policy evaluation and improvement to be parallel, ensuring that better value estimation never inadvertently degrades the policy.

Similarly, Nino Vieillard and the team behind Implicit Q-values take compatibility to its logical extreme. They parametrize the Q-function as a sum of the log-policy and a value function. By construction, the policy becomes the exact softmax of the Q-values, removing the need for a separate Actor network to "track" the Critic.

Finally, work on First-order Sobolev RL by Fabian Schramm suggests that we should match not just the values, but the derivatives of the Bellman target. This "Sobolev" consistency ensures that the Critic provides stable, accurate update directions for the Actor, particularly in continuous control tasks.

Summary

Compatible function approximation works because it respects the functional relationship between acting and evaluating. By constraining the Critic's representation to match the Actor's score function, we transform a difficult estimation problem into a direct geometric update. This synergy ensures that every bit of information the Critic learns is immediately and optimally useful for improving the Actor's behavior.

Papers in this version

History & citationVersion 1 · 2026-10-04

Split into other directions · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_compatible_function_approximation_for_na_3d2cdc74,
  title = {Compatible Function Approximation for Natural Policy Gradients},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/compatible-function-approximation-for-natural-policy-gradients/rdr_3d2cdc74d2f74094afa1326c0f56d1aa},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-21},
}

This exact version (version 1, frozen text):

@misc{lacuna_compatible_function_approximation_for_na_3d2cdc74_v1,
  title = {Compatible Function Approximation for Natural Policy Gradients},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/compatible-function-approximation-for-natural-policy-gradients/rdr_3d2cdc74d2f74094afa1326c0f56d1aa/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-21},
}