Abstract
Offline reinforcement learning aims to train a policy on a pre-recorded and fixed dataset without any additional environment interactions. There are two major challenges in this setting: (1) extrapolation error caused by approximating the value of state-action pairs not well-covered by the training data and (2) distributional shift between behavior and inference policies. One way to tackle these problems is to induce conservatism - i.e., keeping the learned policies closer to the behavioral ones. To achieve this, we build upon recent works on learning policies in latent action spaces and use a special form of Normalizing Flows for constructing a generative model, which we use as a conservative action encoder. This Normalizing Flows action encoder is pre-trained in a supervised manner on the offline dataset, and then an additional policy model - controller in the latent space - is trained via reinforcement learning. This approach avoids querying actions outside of the training dataset and therefore does not require additional regularization for out-of-dataset actions. We evaluate our method on various locomotion and navigation tasks, demonstrating that our approach outperforms recently proposed algorithms with generative action models on a large portion of datasets.
Introduction
Reinforcement Learning (RL) typically relies on an agent learning through trial and error within an environment. However, in Offline RL, the agent must learn exclusively from a fixed, pre-recorded dataset. This setting introduces a significant challenge known as extrapolation error. When an agent selects an action not well-represented in the training data, the value function (critic) often produces over-optimistic estimates, leading to poor performance and instability during deployment.
To mitigate this, researchers employ conservatism, ensuring the learned policy remains close to the behavior policy that generated the data. Previous state-of-the-art methods, such as PLAS, utilize Variational Autoencoders (VAEs) to map a latent space to the action space. However, VAEs typically use Gaussian distributions with infinite support, meaning an agent can still sample "out-of-distribution" actions unless the latent space is manually clipped—a process requiring extensive task-specific tuning and trial-and-error.
Methodology: Conservative Normalizing Flows (CNF)
The proposed Conservative Normalizing Flows (CNF) framework addresses the limitations of VAE-based latent spaces by using Normalizing Flows (NFs) as action encoders. Unlike VAEs, NFs provide bijective, invertible mappings that allow for more precise control over the distribution and density estimation.
The CNF architecture introduces several key innovations to ensure safe action selection:
- Bounded Latent Space: Instead of a standard Gaussian distribution, CNF utilizes a multi-dimensional uniform distribution $U(-1, 1)^n$. This ensures that the latent space itself is inherently bounded.
- Invertible Tanh Layer: An invertible
tanhactivation is applied at the end of the flow to ensure the output actions are naturally bounded within the environment's limits. - Architectural Conservatism: By design, every point in the latent space maps to a valid action supported by the training data. This eliminates the need for manual "clipping" hyperparameters, making the model more robust across different tasks.
The training process consists of two stages. First, the NF action encoder is pre-trained in a supervised manner to maximize the likelihood of actions in the offline dataset. Second, the NF is frozen, and a policy (controller) is trained to select points within the safe latent space using an Advantage Weighted Actor Critic (AWAC) loss.
Experimental Evaluation
The authors evaluated CNF against standard benchmarks in the D4RL suite, including locomotion tasks (Walker2d, Hopper, HalfCheetah) and navigation tasks (Maze2d).
Performance on Locomotion and Navigation
CNF demonstrated superior performance compared to baselines such as PLAS, LAPO, IQL, and AWAC.
- Locomotion: CNF outperformed all generative-model-based baselines on 8 out of 9 locomotion datasets, showing particular strength in
HalfCheetah-v2environments. The average normalized score for CNF was 81.94, significantly higher than IQL (77.25) and PLAS (57.76). - Navigation: In Maze2d tasks, CNF handled long-horizon planning more effectively than previous latent-space models, outperforming baselines in 2 out of 3 environments.
Discussion and Key Insights
A major takeaway is that designing for conservatism through architecture is often more effective than post-hoc regularization. By making it architecturally impossible to select out-of-distribution actions, CNF removes the need for sensitive hyperparameters like the "clipping" value required by VAE methods.
Ablation studies also highlighted the importance of gradient flow. Backpropagating gradients through the Normalizing Flow during policy optimization was found to be more effective than training on static features, as it better guides the policy toward high-reward regions of the supported action space.
While CNF offers significant improvements, it does come with increased architectural complexity compared to VAEs. Future work could explore integrating CNF with other offline RL objectives like CQL or testing its efficacy in higher-dimensional action spaces such as robotic manipulation.
Summary updated