Double Check Your State Before Trusting It: Confidence-Aware Bidirectional Offline Model-Based Imagination

Jiafei Lyu, Xiu Li, Zongqing Lu

Introduction

Offline Reinforcement Learning (RL) is the art of learning a policy from a fixed, pre-collected dataset without any further interaction with the environment. This paradigm is crucial for real-world applications like healthcare or autonomous driving, where "trial and error" exploration is too dangerous or expensive.

However, offline RL faces a core challenge: extrapolation error. If an agent attempts an action not well-represented in the dataset, its value estimates can explode, leading to catastrophic failure. While model-free methods handle this by being extremely conservative (often sacrificing generalization), model-based methods attempt to "imagine" new data using a learned simulator. Unfortunately, if that simulator is even slightly inaccurate, the agent learns from "hallucinations," which can degrade performance even further.

To solve this, researchers introduce CABI (Confidence-Aware Bidirectional Offline Model-Based Imagination), a novel data augmentation framework that boosts offline RL by "double-checking" synthetic data. By training both forward and backward dynamics models, CABI only trusts imagined transitions that are cycle-consistent, ensuring the agent learns exclusively from reliable, high-quality generated trajectories.

The CABI Framework: Double-Checking Imaginations

The authors propose a "double check" mechanism inspired by human caution. The framework consists of three main components designed to safely augment the training dataset:

  1. Bidirectional Dynamics Models: Instead of just predicting the future, the system trains a standard forward model (predicting the next state $s'$ from current state $s$ and action $a$) and a reverse/backward model (predicting the previous state $s$ from $s'$ and $a$).
  2. Bidirectional Rollout Policies: To generate diverse yet realistic actions, the researchers utilize Conditional Variational Autoencoders (CVAEs). These ensure the "imagined" actions stay safely within the distribution of the original dataset.
  3. The Double Check Mechanism: When the forward model predicts a new state $\hat{s}_2$ from $s_1$, the system immediately uses the backward model to "look back" and see if it can recreate $s_1$ from $\hat{s}_2$. If the reconstructed state $\tilde{s}_1$ is very close to the original $s_1$, the transition is deemed reliable and added to the training set.
The multi-step double check mechanism filters out unreliable trajectories by rejecting states that cannot be accurately reversed.

This process is extended over multiple steps. As illustrated above, transitions that drift too far from a reversible path are marked with red crosses and rejected, filtering out compounding errors before they can poison the RL agent's training data.

Key Insights and Results

The researchers evaluated CABI on the D4RL benchmark, a standard suite for offline RL, focusing on challenging Adroit robotic hand tasks, MuJoCo locomotion tasks, and a custom 2D "RiskWorld" environment designed to visualize model failure.

Conservatism Through Cycle Consistency

Most offline RL research focuses on being conservative with value functions (pessimism). CABI introduces conservatism directly into the data generation process. The paper demonstrates that cycle consistency (forward then back) is a powerful proxy for model confidence. If a model cannot "undo" its own prediction, it likely does not understand that region of the state space well.

Visualizing Safety in RiskWorld

In their "RiskWorld" experiment, the authors show that forward-only models often hallucinate transitions into "danger zones" that were not present in the original data. CABI successfully filters these out, keeping the agent's imagined trajectories within safe, known distributions.

Visualizing data generation in RiskWorld. Forward-only (b) and backward-only (c) models drift into the danger zone, while CABI (d) stays safe.

Performance Breakthroughs

CABI acts as a "performance booster" that can be combined with existing model-free algorithms like BCQ and TD3+BC:

  • Adroit Breakthrough: On the notoriously difficult, high-dimensional Adroit tasks (e.g., twirling a pen, opening a door), CABI+BCQ achieved a total score of 607.0, dwarfing vanilla BCQ (476.7). Notably, it crushed existing model-based baselines like MOPO (6.4) and COMBO (59.1), which typically fail on these tasks due to compounding model errors.
  • MuJoCo Consistency: On MuJoCo locomotion tasks, CABI+TD3_BC outperformed the base TD3_BC algorithm on 14 out of 15 datasets, proving its broad utility.
  • Ablation Proof: The authors tested a version without the double-check (called BOMI). BOMI performed worse than CABI, proving that simply having bidirectional models isn't enough—the cycle-consistency filter is essential.

Limitations and Future Work

The primary limitation of CABI is its computational cost. Training four separate models (forward/backward dynamics and forward/backward rollout policies) and performing the double-check during data generation adds significant overhead compared to pure model-free methods.

For future work, the authors suggest exploring CABI in online RL settings to see if this bidirectional "imagination" can improve sample efficiency during active exploration. They also aim to investigate if the framework can be further optimized to reduce the training burden while maintaining its robust safety guarantees.

Summary updated

Abstract

The learned policy of model-free offline reinforcement learning (RL) methods is often constrained to stay within the support of datasets to avoid possible dangerous out-of-distribution actions or states, making it challenging to handle out-of-support region. Model-based RL methods offer a richer dataset and benefit generalization by generating imaginary trajectories with either trained forward or reverse dynamics model. However, the imagined transitions may be inaccurate, thus downgrading the performance of the underlying offline RL method. In this paper, we propose to augment the offline dataset by using trained bidirectional dynamics models and rollout policies with double check. We introduce conservatism by trusting samples that the forward model and backward model agree on. Our method, confidence-aware bidirectional offline model-based imagination, generates reliable samples and can be combined with any model-free offline RL method. Experimental results on the D4RL benchmarks demonstrate that our method significantly boosts the performance of existing model-free offline RL algorithms and achieves competitive or better scores against baseline methods.
BibTeX
@inproceedings{lyu2022double,
  author = {Jiafei Lyu and Xiu Li and Zongqing Lu},
  title = {{Double Check Your State Before Trusting It: Confidence-Aware Bidirectional Offline Model-Based Imagination}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2022},
  url = {https://openreview.net/forum?id=wmVgFh_mgGL},
}
Figure