Abstract
Modern Deep Reinforcement Learning (RL) algorithms require estimates of the maximal Q-value, which are difficult to compute in continuous domains with an infinite number of possible actions. In this work, we introduce a new update rule for online and offline RL which directly models the maximal value using Extreme Value Theory (EVT), drawing inspiration from economics. By doing so, we avoid computing Q-values using out-of-distribution actions which is often a substantial source of error. Our key insight is to introduce an objective that directly estimates the optimal soft-value functions (LogSumExp) in the maximum entropy RL setting without needing to sample from a policy. Using EVT, we derive our Extreme Q-Learning framework and consequently online and, for the first time, offline MaxEnt Q-learning algorithms, that do not explicitly require access to a policy or its entropy. Our method obtains consistently strong performance in the D4RL benchmark, outperforming prior works by 10+ points on the challenging Franka Kitchen tasks while offering moderate improvements over SAC and TD3 on online DM Control tasks.
Introduction
In Reinforcement Learning (RL), the core challenge in continuous action spaces is the "max" operation within the Bellman equation. To determine the optimal value for a given state, an agent must theoretically evaluate an infinite number of possible actions. Standard approaches, such as Soft Actor-Critic (SAC), attempt to bypass this by utilizing an "actor" (a policy network) to approximate the best action.
However, this indirect estimation becomes a significant liability in offline RL, where the agent learns from a fixed dataset without environment interaction. In these scenarios, the actor often suggests "out-of-distribution" (OOD) actions. Because the critic has no data for these actions, it frequently produces wildly over-optimistic value estimates, leading to training instability and poor performance. Extreme Q-Learning (X-QL) addresses this by computing the "soft-max" (LogSumExp) of Q-values directly from the data, avoiding OOD sampling entirely.
Methodology: The Gumbel Regression Framework
X-QL introduces two major innovations rooted in Extreme Value Theory (EVT):
1. The Gumbel Error Model
The authors observe that Bellman errors in deep RL are typically skewed rather than Gaussian. Since RL is fundamentally concerned with the maximum of random variables, EVT suggests that these errors should follow a Gumbel distribution. This shift acknowledges that the upper tail of the distribution—representing the best possible outcomes—is what matters most for value estimation.
2. Gumbel Regression Loss
By applying the Gumbel distribution to the regression objective, the authors derive a new loss function: $$\mathcal{L}(h) = \mathbb{E}[e^{(x-h)/\beta} - (x-h)/\beta - 1]$$
Minimizing this loss directly recovers the LogSumExp (the soft-maximum) of the underlying values. When applied to the Bellman update, this allows the framework to perform Maximum Entropy (MaxEnt) RL—known for its robustness and exploratory benefits—while treating the soft-max as a simple regression problem. In the offline setting, this means the value of the next state is estimated using only actions present in the dataset, providing a natural bridge to conservative methods without explicit OOD penalties.
Experimental Evaluation
The authors evaluated X-QL across the standard D4RL benchmark and the DeepMind (DM) Control Suite.
- Franka Kitchen: X-QL achieved state-of-the-art results, outperforming prior methods like IQL and CQL by over 10 points on these complex, long-horizon robotic manipulation tasks.
- AntMaze: The method showed significant success in navigation tasks requiring trajectory "stitching," reaching near-perfect scores (e.g., 99.6 on
umaze-v0) after online fine-tuning. - Online Performance: In online settings, X-QL variants (X-SAC and X-TD3) matched or slightly exceeded the performance of their vanilla counterparts on DM Control tasks, demonstrating the framework's versatility across learning paradigms.
Key Insights
MaxEnt without the "Ent"
Traditionally, MaxEnt RL requires a policy network to calculate $\log \pi(a|s)$ to regularize the objective. X-QL demonstrates that the Gumbel distribution implicitly handles this regularization. This allows practitioners to reap the benefits of MaxEnt RL without the computational overhead of explicit entropy calculations or nested sampling loops.
Implicit Conservatism
In offline RL, X-QL is naturally conservative. By fitting the soft-max value to the existing data distribution, it avoids "hallucinating" high values for unseen actions. This makes the algorithm inherently more stable than traditional actor-critic methods that rely on extrapolation.
Limitations and Future Work
While powerful, X-QL requires careful handling of the exponential term in the Gumbel loss ($e^{(x-h)/\beta}$), which can lead to gradient instability. The authors mitigate this using max-normalization and gradient clipping. Additionally, the temperature parameter $\beta$ remains sensitive and requires environment-specific tuning. Future research may explore hybrid Gumbel-Gaussian models to better characterize error propagation in highly stochastic environments.
Summary updated