Monotonic Policy Improvement under Function Approximation

This direction is no longer present in the current build. Its last version stays readable and citable here.

Version 1 of 130 papers31 conceptsfrozen 2026-10-04

In the idealized world of Reinforcement Learning (RL) theory, agents possess a "God's-eye view" of their environment. They operate in perfect Markov Decision Processes (MDPs) and calculate exact value functions. In this vacuum, algorithms like Policy Gradient are mathematically guaranteed to improve.

However, real-world RL—from robotics to large language models—operates in a state of constant approximation. Agents face noisy observations, use neural networks to guess value functions, and work with limited data. This creates a "theoretical blind spot": when the value function (the critic) is even slightly wrong, the policy update (the actor) can become reckless, leading to performance degradation rather than steady improvement.

A growing body of research is bridging this gap by developing frameworks that guarantee monotonic policy improvement—ensuring the agent gets better at every step—even when its internal models are imperfect.

The Objective Mismatch Problem

The core challenge in modern Actor-Critic methods is that the actor and the critic are often optimized for different things. The critic tries to minimize mean-squared error in value estimation, while the actor tries to maximize expected return. If the critic's errors happen to be in regions of the state space critical for decision-making, the actor will follow a "false light."

Sharan Vaswani, Nicolas Le Roux, and their colleagues address this in Decision-Aware Actor-Critic with Function Approximation and Theoretical Guarantees. They argue that the critic should be "decision-aware"—explicitly optimized to help the actor make better choices. By deriving a joint lower bound on policy improvement, they create a training objective where the critic’s estimation error is mathematically linked to the actor's update, guaranteeing improvement despite using function approximators like neural networks.

Stability Through Contractions and Bounds

To trust an update, we first need to know how far our estimates can drift. In off-policy learning—where an agent learns from data gathered by a different policy—estimates can easily diverge.

Assaf Hallak, Aviv Tamar, and Shie Mannor provided a foundational piece of this puzzle in Emphatic TD Bellman Operator is a Contraction. By proving that the operator underlying Emphatic Temporal Difference (ETD) learning is a "contraction," they showed that the algorithm consistently pulls estimates toward the truth. This provides the formal error bounds necessary to predict how accurate a converged solution will actually be.

Similarly, in the realm of partially observable environments (POMDPs), where the agent cannot see the full state, Kamyar Azizzadenesheli, Yisong Yue, and Animashree Anandkumar highlight in Policy Gradient in Partially Observable Environments: Approximation and Convergence that standard algorithms lack monotonic guarantees. Their work emphasizes that without accounting for the "noise" of partial observability, policy updates can become unstable.

Functional Mirror Ascent

How do we design a general recipe for stable updates? Sharan Vaswani and Matthieu Geist proposed a unified framework called Functional Mirror Ascent for Policy Gradient (FMA-PG) in A general class of surrogate functions for stable and efficient reinforcement learning.

FMA-PG uses surrogate functions to ensure that even when action-values are estimated from noisy samples, the resulting policy update remains safe. This framework allows for the derivation of more robust versions of popular algorithms, like a "softmax" PPO, which maintains theoretical safety even when reusing old data.

Balancing Trust and Innovation

In offline settings, where an agent learns from a fixed dataset, the risk of "over-optimizing" against a noisy value function is high. Sungryull Sohn, Ofir Nachum, and their team developed BRPO: Batch Residual Policy Optimization to solve this.

BRPO works by blending a new "candidate" policy with the existing "behavior" policy. The mechanism uses a state-action-dependent confidence factor:

  1. High Confidence: If the advantage function estimate is reliable, the agent trusts the new policy.
  2. Low Confidence: If the data is sparse or noisy, the agent sticks to the safe, known behavior.

This "residual" approach provides a principled way to innovate without violating the performance lower bounds that keep the agent from failing.

Moving Beyond Exactness

The shift across these papers—from Hepeng Li’s analytical update rules in An Analytical Update Rule for General Policy Optimization to David Brandfonbrener’s quantile filtering in Quantile Filtered Imitation Learning—is a move toward robustness by design.

By acknowledging that critics will be wrong and data will be noisy, these researchers are building RL algorithms that don't just work in the "God's-eye" simulations of the past, but remain stable and improving in the messy, approximated reality of the present.

Papers in this version

History & citationVersion 1 · 2026-10-04

No longer present in the current build · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_monotonic_policy_improvement_under_funct_dd69b9b6,
  title = {Monotonic Policy Improvement under Function Approximation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/monotonic-policy-improvement-under-function-approximation/rdr_dd69b9b6b3c94ce296107a44f943e9b4},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-21},
}

This exact version (version 1, frozen text):

@misc{lacuna_monotonic_policy_improvement_under_funct_dd69b9b6_v1,
  title = {Monotonic Policy Improvement under Function Approximation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/monotonic-policy-improvement-under-function-approximation/rdr_dd69b9b6b3c94ce296107a44f943e9b4/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-21},
}