Introduction
In Reinforcement Learning (RL), agents learn to maximize rewards through trial and error. Fitted Q-iteration (FQI) is a popular "batch" RL method known for being highly sample-efficient and stable. However, it has a major Achilles' heel: it struggles with continuous action spaces.
Standard FQI relies on a "greedy" policy improvement step, requiring the algorithm to find the specific action that maximizes the Q-function for every state. In discrete settings (like choosing to move "left" or "right"), this is trivial. But in continuous settings (like determining the precise torque of a robotic arm), finding this maximum requires solving a complex optimization problem at every single iteration. This is not only computationally expensive but often introduces "optimization bias," leading to unstable learning and "jerky," non-smooth movements that could easily damage real-world hardware.
To solve this, researchers introduce LAWER (Locally-Advantage-WEighted Regression), an algorithm that makes FQI practical for continuous, high-dimensional action spaces by replacing the expensive greedy action selection with a "soft-greedy" advantage-weighted regression.
The LAWER Methodology
The authors propose a clever workaround: instead of searching for the absolute best action through hard optimization, they use a soft-greedy action selection based on Advantage Weighted Regression (AWR).
- Soft-Greedy Policy: Instead of a hard "max" operator, the algorithm uses a stochastic policy where actions are weighted by their "advantage"—a measure of how much better a specific action is compared to the average action in that state.
- The Regression Trick: Using Jensen’s inequality, the authors prove that estimating the value function for this soft-greedy policy can be mathematically transformed into a simple weighted regression problem. This entirely eliminates the need for expensive optimization loops, such as the Cross-Entropy (CE) Method, during training.
- Locally-Advantage-Weighted Regression (LAWER): The theory is implemented using Locally Weighted Regression (LWR) as the function approximator. LAWER maintains a local estimate of the mean and variance of the advantage function to weight the incoming data points, allowing it to smoothly approximate the optimal policy.
Key Insights and Ideas
The core "aha moment" of this research is that we do not actually need to find the maximum action to improve a policy. By weighting existing experiences by their relative performance (advantage), we can "regress" toward the optimal value function much more efficiently.
Furthermore, smoothness matters. Because LAWER uses a soft-max approach rather than a hard "winner-take-all" max, the resulting control trajectories are significantly smoother. This is a critical requirement for robotics, where jerky movements cause mechanical wear and tear. Finally, by using a weighted average (regression), LAWER bypasses the "max" bias that standard FQI suffers from, making it much more robust to noise.
Experimental Setup and Key Results
The researchers evaluated LAWER on three classic control benchmarks, modified for continuous actions: the Pendulum Swing-up, the Acrobot Swing-up, and a Dynamic Puddle-world navigation task (tested in both 2D and 3D to evaluate scalability).
The results demonstrated significant improvements across three main axes:
- Speed: LAWER is remarkably faster than FQI methods that rely on Cross-Entropy optimization. For example, in the Pendulum task, LAWER took approximately 2,780 seconds to train, compared to roughly 59,900 seconds for the tree-based FQI—a massive 20x speedup.
- Performance: Across all tasks, LAWER achieved comparable or superior total rewards compared to neural-network and tree-based FQI variants, all while maintaining much higher computational efficiency.
- Scalability: In the high-dimensional 3D Puddle-world task, the baseline (CE-Tree) struggled to find a good policy and produced very jerky, almost random movements. LAWER, however, easily found a smooth, goal-directed policy.
Limitations and Future Work
While LAWER presents a major step forward for continuous FQI, the authors note a few areas for future improvement. The algorithm relies on distance kernels (like Gaussian kernels), and the "bandwidth" of these kernels must be chosen manually. In very high-dimensional state spaces, tuning these parameters becomes difficult. Additionally, the current implementation assumes a uniform distance measure across the entire state space, which might not be ideal for complex tasks where certain dimensions are more critical than others.
Future work may involve combining the AWR approach with other function approximators, such as regression trees, to handle even more complex and varied state spaces without requiring uniform distance metrics.
Summary updated