Let Offline RL Flow: Training Conservative Agents in the Latent Space of Normalizing Flow

Dmitry Akimov, Vladislav Kurenkov, Alexander Nikulin, Denis Tarasov, Sergey KolesnikovICLR 2023 · OpenReview

TL;DR

This paper introduces Conservative Normalizing Flows (CNF), a novel offline reinforcement learning (RL) method that trains agents in a learned latent action space. By using normalizing flows to create a bounded latent action space, CNF effectively prevents the agent from exploring out-of-distribution actions, a common pitfall in offline RL, leading to improved performance on benchmark tasks.

What Problem Does This Paper Address?

Offline Reinforcement Learning (ORL) aims to train agents using only pre-collected datasets, without further interaction with the environment. This setting presents two major challenges:

  1. Extrapolation Error: When the agent encounters states or actions not well-represented in the dataset, value estimation can become inaccurate, leading to overestimation bias. This is particularly problematic when the agent tries to infer the value of actions that were never taken in the training data.
  2. Distributional Shift: The policy being trained might deviate significantly from the behavior policy that generated the dataset, leading to unreliable value estimates.

Existing methods try to mitigate these issues through conservatism, essentially forcing the learned policy to stay "close" to the behavior policy. However, some approaches, like those using Variational Autoencoders (VAEs) for latent action spaces, require ad-hoc clipping of the latent space to prevent issues, which can discard potentially useful actions.

Methods

The core of CNF lies in its use of Normalizing Flows (NFs) to construct a conservative latent action space. Here's a breakdown:

  • Normalizing Flows as Action Encoder: Instead of VAEs, CNF employs NFs to learn a mapping from the original action space to a latent space. NFs are invertible, differentiable generative models that allow for exact likelihood computation.
  • Bounded Latent Space: A key innovation is the use of an invertible tanh activation after the last layer of the NF model. This ensures that the latent action space is bounded (e.g., within (-1, +1)^n), naturally preventing the agent from generating actions outside the learned distribution. This also allows for the use of a uniform distribution as the base distribution in the latent space, which is more amenable to exploration within the bounded region.
  • Two-Phase Training:
    1. Pre-training: The NF action encoder is first trained in a supervised manner on the offline dataset to learn a mapping from states and actions to the latent space. This phase aims to capture the data distribution effectively.
  • Policy Optimization: A latent policy (controller) is then trained using RL algorithms (specifically, an adaptation of Advantage Weighted Actor-Critic) within this bounded latent action space. The NF encoder is frozen during this phase.

This approach avoids the need for manual clipping of the latent space, which was a workaround in previous VAE-based methods, and ensures that the agent's actions remain within the support of the training data distribution.

Datasets

The paper evaluates CNF on the following D4RL (Datasets for Deep Data-Driven Reinforcement Learning) benchmark datasets:

  • Locomotion: Walker2d-v2, Hopper-v2, and HalfCheetah-v2 (in medium, medium-replay, and medium-expert variations).
  • Maze2d: umaze-v1, medium-v1, and large-v1.

Key Insights and Ideas

  • Normalizing Flows for Bounded Latent Spaces: The paper demonstrates that NFs, when combined with an invertible tanh activation, provide a principled way to create a bounded latent action space, inherently addressing the extrapolation error problem without ad-hoc clipping.
  • Uniform vs. Normal Latent Distributions: While both normal and uniform base distributions can be used with NFs, the paper's experiments suggest that a uniform latent distribution, enabled by the bounded nature of the NF output, leads to better performance. This is because it allows the policy to utilize the entire latent space without the risk of generating out-of-distribution actions.
  • Implicit Conservatism: By operating within a learned, bounded latent space that reflects the training data distribution, CNF implicitly enforces conservatism, reducing the need for explicit regularization terms that penalize out-of-distribution actions.
  • Pre-training is Crucial: The supervised pre-training of the NF action encoder is shown to be essential for learning a good latent representation that can then be leveraged by the RL policy.

Key Results

  • CNF outperforms recently proposed generative model-based offline RL algorithms (PLAS, LAPO) on a significant portion of the D4RL locomotion and maze2d datasets.
  • Specifically, CNF achieves better normalized scores across 8 out of 9 locomotion datasets and 2 out of 3 maze2d datasets compared to AWAC, IQL, and PLAS.
  • Ablation studies show that training a latent policy without clipping in a normal latent space leads to substantially worse performance, highlighting the importance of the bounded latent space provided by CNF.
  • Training the controller directly in the latent space (without backpropagating gradients through the NF encoder) yields slightly lower performance, suggesting that leveraging the gradients from the NF model is beneficial for policy optimization.

Limitations and Future Work

  • Computational Cost: While NFs offer advantages, their training can be computationally intensive. The paper mentions the use of Adam optimizer and grid-search for hyperparameters, indicating a need for careful tuning.
  • Generalizability of Latent Space: The paper focuses on locomotion and maze tasks. Further research could explore the effectiveness of CNF on a wider range of domains and task complexities.
  • Orthogonal Direction: The authors note that their work explores a different direction than LAPO, which jointly trains the action encoder and RL agent. Future work could investigate hybrid approaches.
  • Exploration in Latent Space: While CNF prevents out-of-distribution actions, further research could explore how to encourage more effective exploration within the learned latent action space to discover optimal policies.

Summary updated

Abstract

Offline reinforcement learning aims to train a policy on a pre-recorded and fixed dataset without any additional environment interactions. There are two major challenges in this setting: (1) extrapolation error caused by approximating the value of state-action pairs not well-covered by the training data and (2) distributional shift between behavior and inference policies. One way to tackle these problems is to induce conservatism - i.e., keeping the learned policies closer to the behavioral ones. To achieve this, we build upon recent works on learning policies in latent action spaces and use a special form of normalizing flow for constructing a generative model, which we use as a conservative action encoder. This normalizing flow action encoder is pre-trained in a supervised manner on the offline dataset, and then an additional policy model - controller in the latent space - is trained via reinforcement learning. This approach avoids querying actions outside of the training dataset and therefore does not require additional regularization for out-of-dataset actions. We evaluate our method on various locomotion and navigation tasks, demonstrating that our approach outperforms recently proposed algorithms with generative action models on a large portion of datasets.
BibTeX
@inproceedings{akimov2023let,
  author = {Dmitry Akimov and Vladislav Kurenkov and Alexander Nikulin and Denis Tarasov and Sergey Kolesnikov},
  title = {{Let Offline RL Flow: Training Conservative Agents in the Latent Space of Normalizing Flow}},
  booktitle = {International Conference on Learning Representations},
  year = {2023},
  url = {https://openreview.net/forum?id=1Wo0vqaZ8WJ},
}
Figure