Fitted Q-iteration by Advantage Weighted Regression

Gerhard Neumann, Jan Peters

Introduction

In Reinforcement Learning (RL), agents learn to maximize rewards through trial and error. Fitted Q-iteration (FQI) is a popular "batch" RL method known for being highly sample-efficient and stable. However, it has a major Achilles' heel: it struggles with continuous action spaces.

Standard FQI relies on a "greedy" policy improvement step, requiring the algorithm to find the specific action that maximizes the Q-function for every state. In discrete settings (like choosing to move "left" or "right"), this is trivial. But in continuous settings (like determining the precise torque of a robotic arm), finding this maximum requires solving a complex optimization problem at every single iteration. This is not only computationally expensive but often introduces "optimization bias," leading to unstable learning and "jerky," non-smooth movements that could easily damage real-world hardware.

To solve this, researchers introduce LAWER (Locally-Advantage-WEighted Regression), an algorithm that makes FQI practical for continuous, high-dimensional action spaces by replacing the expensive greedy action selection with a "soft-greedy" advantage-weighted regression.

The LAWER Methodology

The authors propose a clever workaround: instead of searching for the absolute best action through hard optimization, they use a soft-greedy action selection based on Advantage Weighted Regression (AWR).

  1. Soft-Greedy Policy: Instead of a hard "max" operator, the algorithm uses a stochastic policy where actions are weighted by their "advantage"—a measure of how much better a specific action is compared to the average action in that state.
  2. The Regression Trick: Using Jensen’s inequality, the authors prove that estimating the value function for this soft-greedy policy can be mathematically transformed into a simple weighted regression problem. This entirely eliminates the need for expensive optimization loops, such as the Cross-Entropy (CE) Method, during training.
  3. Locally-Advantage-Weighted Regression (LAWER): The theory is implemented using Locally Weighted Regression (LWR) as the function approximator. LAWER maintains a local estimate of the mean and variance of the advantage function to weight the incoming data points, allowing it to smoothly approximate the optimal policy.

Key Insights and Ideas

The core "aha moment" of this research is that we do not actually need to find the maximum action to improve a policy. By weighting existing experiences by their relative performance (advantage), we can "regress" toward the optimal value function much more efficiently.

Furthermore, smoothness matters. Because LAWER uses a soft-max approach rather than a hard "winner-take-all" max, the resulting control trajectories are significantly smoother. This is a critical requirement for robotics, where jerky movements cause mechanical wear and tear. Finally, by using a weighted average (regression), LAWER bypasses the "max" bias that standard FQI suffers from, making it much more robust to noise.

Comparison of learning curves and control trajectories. LAWER achieves higher rewards and significantly smoother control signals compared to Cross-Entropy (CE) methods.

Experimental Setup and Key Results

The researchers evaluated LAWER on three classic control benchmarks, modified for continuous actions: the Pendulum Swing-up, the Acrobot Swing-up, and a Dynamic Puddle-world navigation task (tested in both 2D and 3D to evaluate scalability).

The results demonstrated significant improvements across three main axes:

  • Speed: LAWER is remarkably faster than FQI methods that rely on Cross-Entropy optimization. For example, in the Pendulum task, LAWER took approximately 2,780 seconds to train, compared to roughly 59,900 seconds for the tree-based FQI—a massive 20x speedup.
  • Performance: Across all tasks, LAWER achieved comparable or superior total rewards compared to neural-network and tree-based FQI variants, all while maintaining much higher computational efficiency.
  • Scalability: In the high-dimensional 3D Puddle-world task, the baseline (CE-Tree) struggled to find a good policy and produced very jerky, almost random movements. LAWER, however, easily found a smooth, goal-directed policy.
Performance in the 3D Puddle-world task. LAWER maintains smooth control across multiple dimensions ($u_1, u_2, u_3$), whereas the CE Tree baseline produces erratic movements.

Limitations and Future Work

While LAWER presents a major step forward for continuous FQI, the authors note a few areas for future improvement. The algorithm relies on distance kernels (like Gaussian kernels), and the "bandwidth" of these kernels must be chosen manually. In very high-dimensional state spaces, tuning these parameters becomes difficult. Additionally, the current implementation assumes a uniform distance measure across the entire state space, which might not be ideal for complex tasks where certain dimensions are more critical than others.

Future work may involve combining the AWR approach with other function approximators, such as regression trees, to handle even more complex and varied state spaces without requiring uniform distance metrics.

Summary updated

Abstract

Recently, fitted Q-iteration (FQI) based methods have become more popular due to their increased sample efficiency, a more stable learning process and the higher quality of the resulting policy. However, these methods remain hard to use for continuous action spaces which frequently occur in real-world tasks, e.g., in robotics and other technical applications. The greedy action selection commonly used for the policy improvement step is particularly problematic as it is expensive for continuous actions, can cause an unstable learning process, introduces an optimization bias and results in highly non-smooth policies unsuitable for real-world systems. In this paper, we show that by using a soft-greedy action selection the policy improvement step used in FQI can be simplified to an inexpensive advantage-weighted regression. With this result, we are able to derive a new, computationally efficient FQI algorithm which can even deal with high dimensional action spaces.
BibTeX
@inproceedings{neumann2008fitted,
  author = {Gerhard Neumann and Jan Peters},
  title = {{Fitted Q-iteration by Advantage Weighted Regression}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2008},
}
Figure