Gradient Based Value Function Approximation

21 concepts13 papers1 proposal

In standard reinforcement learning (RL), an agent typically learns a scalar value function—such as $V(s)$ or $Q(s,a)$—which maps a state or state-action pair to the expected cumulative future reward. To optimize a policy, the agent looks at how these scalar values change across different actions or states.

However, this traditional approach introduces a fundamental mismatch: policies are driven by marginal changes (gradients), yet our models are trained to predict absolute totals (scalars). Gradient-based value function approximation flips this paradigm by training critics to directly estimate the derivatives of the value function with respect to states, goals, or policy parameters.

The Core Intuition: Why Learn Gradients?

To understand why learning gradients directly is advantageous, consider what happens when you shift a value function up or down by a constant offset. The absolute values change completely, but the optimal policy remains identical because the relative differences between states are preserved.

When we force a neural network to approximate a scalar value function, a significant portion of its capacity is consumed trying to resolve these global, irrelevant offsets. As Dayan et al. (1995) early noted, shifting the representation from absolute values to spatial derivatives focuses the learning process entirely on the local direction of improvement.

Yang (2023) and Lee et al. (2022) leverage this property to show that fitting value gradients directly drastically reduces the amount of simulation data required to find effective control policies. Because optimal decisions depend strictly on marginal changes, matching the gradient captures the essential information for control far more efficiently than global cost estimation.

The Gradient Bellman Equation

If we choose to learn gradients directly, how do we construct a bootstrapping target without relying on a noisy scalar value function? The breakthrough that makes this practical is that gradients themselves satisfy a recursive, Bellman-like structure.

Tosatto et al. (2022) formalized this by deriving the Gradient Bellman Equation. They demonstrated that policy gradients can be expressed as a discounted accumulation of immediate gradients along a trajectory, allowing them to be estimated recursively using standard temporal-difference (TD) methods. Similarly, Ni et al. (2022) showed that the gradient of the action-value function satisfies a recursive structure that can be solved via iterative regression.

Standard TD Learning:  V(s)  = R(s) + γ * V(s')
Gradient TD Learning: ∇V(s) = ∇R(s) + γ * ∇V(s') * [Transition Jacobian]

This recursive formulation solves a notorious problem in off-policy RL: variance explosion. Traditional off-policy policy gradient methods rely on importance sampling ratios to correct for the mismatch between the behavior policy and the target policy, which often leads to catastrophic variance. By utilizing a specialized "gradient critic" that tracks derivatives recursively, the algorithm predicts the gradient that would occur under the target policy directly from off-policy data, bypassing importance sampling ratios entirely (Xu et al. (2021); Tosatto et al. (2022)).

Sobolev Learning and Landscape Geometry

In continuous state spaces or goal-conditioned tasks, supervising a model with gradients provides a much richer geometric signal than scalar rewards alone. In machine learning, training a network on both a function's scalar values and its derivatives is known as Sobolev learning.

Alboni et al. (2024) demonstrate that training a critic network on both the scalar value function and its spatial gradient provides a highly detailed map of the landscape's local shape. This dual supervision extracts significantly more information from each optimization step, accelerating sample efficiency in complex robotic trajectory optimization.

In goal-conditioned reinforcement learning, where rewards are often sparse, Levine et al. (2023) show that supervising a value function with the gradient of the target value with respect to the goal provides an exceptionally dense learning signal. Instead of receiving a single scalar feedback point, the agent learns the local shape of the entire goal landscape, providing a volume of supervision proportional to the dimensionality of the goal space.

Smoothness and Stability in Continuous Control

Beyond sample efficiency, value gradient learning fundamentally alters the stability of the resulting controller. In continuous physical systems like autonomous driving or robotics, standard RL controllers often suffer from high-frequency "jitter." This happens because small, high-frequency approximation errors in a scalar value function become amplified when the policy differentiates the value landscape to select an action.

Huang et al. (2016) and Liu et al. (2017) address this by training critics to estimate the derivatives of the value function with respect to system states directly. Because the policy is updated using a smooth, explicitly learned gradient landscape rather than a numerically differentiated scalar landscape, the resulting control signals are significantly more stable, ensuring smoother transitions and faster stabilization during complex maneuvers.

Algorithmic Extensions

The mathematical consistency of value gradients allows them to integrate cleanly into other advanced optimization frameworks:

  • Momentum Acceleration: Weng et al. (2021) demonstrate that the difference between consecutive applications of the empirical Bellman operator can be used to approximate a gradient direction. This allows algorithms to leverage Nesterov’s and Polyak’s momentum acceleration schemes directly within temporal difference frameworks.
  • Differential Equations: Kerimkulov et al. (2020) prove that updating the gradient of the value function directly provides an alternative, exponentially stable path to solving Hamilton-Jacobi-Bellman equations for controlled diffusions.
  • Distributional Tasks: Zhou et al. (2022) adapt these principles to policy optimization over finite Markov chains, computing the gradients of average rewards to efficiently isolate quasi-stationary distributions.

Key Foundational Work

Lateral control for Autonomous land Vehicles via Dual Heuristic Programming (2016) — contributed 2 concept elements to this direction.
Improving Policies without Measuring Merits (1995) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_gradient_based_value_function_approximat_410e49f7,
  title = {gradient-based value function approximation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/gradient-based-value-function-approximation/rdr_410e49f711a443da91e21edf89466c20},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_gradient_based_value_function_approximat_410e49f7_v1,
  title = {gradient-based value function approximation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/gradient-based-value-function-approximation/rdr_410e49f711a443da91e21edf89466c20/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake