TL;DR
This paper introduces Conservative Normalizing Flows (CNF), a novel offline reinforcement learning (RL) method that trains agents in a learned latent action space. By using normalizing flows to create a bounded latent action space, CNF effectively prevents the agent from exploring out-of-distribution actions, a common pitfall in offline RL, leading to improved performance on benchmark tasks.
What Problem Does This Paper Address?
Offline Reinforcement Learning (ORL) aims to train agents using only pre-collected datasets, without further interaction with the environment. This setting presents two major challenges:
- Extrapolation Error: When the agent encounters states or actions not well-represented in the dataset, value estimation can become inaccurate, leading to overestimation bias. This is particularly problematic when the agent tries to infer the value of actions that were never taken in the training data.
- Distributional Shift: The policy being trained might deviate significantly from the behavior policy that generated the dataset, leading to unreliable value estimates.
Existing methods try to mitigate these issues through conservatism, essentially forcing the learned policy to stay "close" to the behavior policy. However, some approaches, like those using Variational Autoencoders (VAEs) for latent action spaces, require ad-hoc clipping of the latent space to prevent issues, which can discard potentially useful actions.
Methods
The core of CNF lies in its use of Normalizing Flows (NFs) to construct a conservative latent action space. Here's a breakdown:
- Normalizing Flows as Action Encoder: Instead of VAEs, CNF employs NFs to learn a mapping from the original action space to a latent space. NFs are invertible, differentiable generative models that allow for exact likelihood computation.
- Bounded Latent Space: A key innovation is the use of an invertible tanh activation after the last layer of the NF model. This ensures that the latent action space is bounded (e.g., within
(-1, +1)^n), naturally preventing the agent from generating actions outside the learned distribution. This also allows for the use of a uniform distribution as the base distribution in the latent space, which is more amenable to exploration within the bounded region. - Two-Phase Training:
- Pre-training: The NF action encoder is first trained in a supervised manner on the offline dataset to learn a mapping from states and actions to the latent space. This phase aims to capture the data distribution effectively.
- Policy Optimization: A latent policy (controller) is then trained using RL algorithms (specifically, an adaptation of Advantage Weighted Actor-Critic) within this bounded latent action space. The NF encoder is frozen during this phase.
This approach avoids the need for manual clipping of the latent space, which was a workaround in previous VAE-based methods, and ensures that the agent's actions remain within the support of the training data distribution.
Datasets
The paper evaluates CNF on the following D4RL (Datasets for Deep Data-Driven Reinforcement Learning) benchmark datasets:
- Locomotion: Walker2d-v2, Hopper-v2, and HalfCheetah-v2 (in medium, medium-replay, and medium-expert variations).
- Maze2d: umaze-v1, medium-v1, and large-v1.
Key Insights and Ideas
- Normalizing Flows for Bounded Latent Spaces: The paper demonstrates that NFs, when combined with an invertible tanh activation, provide a principled way to create a bounded latent action space, inherently addressing the extrapolation error problem without ad-hoc clipping.
- Uniform vs. Normal Latent Distributions: While both normal and uniform base distributions can be used with NFs, the paper's experiments suggest that a uniform latent distribution, enabled by the bounded nature of the NF output, leads to better performance. This is because it allows the policy to utilize the entire latent space without the risk of generating out-of-distribution actions.
- Implicit Conservatism: By operating within a learned, bounded latent space that reflects the training data distribution, CNF implicitly enforces conservatism, reducing the need for explicit regularization terms that penalize out-of-distribution actions.
- Pre-training is Crucial: The supervised pre-training of the NF action encoder is shown to be essential for learning a good latent representation that can then be leveraged by the RL policy.
Key Results
- CNF outperforms recently proposed generative model-based offline RL algorithms (PLAS, LAPO) on a significant portion of the D4RL locomotion and maze2d datasets.
- Specifically, CNF achieves better normalized scores across 8 out of 9 locomotion datasets and 2 out of 3 maze2d datasets compared to AWAC, IQL, and PLAS.
- Ablation studies show that training a latent policy without clipping in a normal latent space leads to substantially worse performance, highlighting the importance of the bounded latent space provided by CNF.
- Training the controller directly in the latent space (without backpropagating gradients through the NF encoder) yields slightly lower performance, suggesting that leveraging the gradients from the NF model is beneficial for policy optimization.
Limitations and Future Work
- Computational Cost: While NFs offer advantages, their training can be computationally intensive. The paper mentions the use of Adam optimizer and grid-search for hyperparameters, indicating a need for careful tuning.
- Generalizability of Latent Space: The paper focuses on locomotion and maze tasks. Further research could explore the effectiveness of CNF on a wider range of domains and task complexities.
- Orthogonal Direction: The authors note that their work explores a different direction than LAPO, which jointly trains the action encoder and RL agent. Future work could investigate hybrid approaches.
- Exploration in Latent Space: While CNF prevents out-of-distribution actions, further research could explore how to encourage more effective exploration within the learned latent action space to discover optimal policies.
Summary updated