Backward Imitation and Forward Reinforcement Learning via Bi-directional Model Rollouts

Yuxin Pan, Fangzhen Lin

Abstract

Sample efficiency remains a primary bottleneck in applying reinforcement learning (RL) to complex continuous control tasks. While Model-Based Reinforcement Learning (MBRL) improves efficiency by learning a world model, existing bi-directional rollout methods often fail to distinguish between the utility of forward and backward trajectories. This paper introduces Backward Imitation and Forward Reinforcement Learning (BIFRL), a framework that treats backward model rollouts from high-value states as expert demonstrations. By combining imitation learning on these backward traces with traditional forward RL exploration, BIFRL significantly accelerates convergence and improves sample efficiency across standard benchmarks.

Introduction

Reinforcement learning often requires millions of interactions with an environment to achieve stable performance, a requirement that is frequently prohibitive in real-world robotics. Model-Based RL (MBRL) addresses this by learning a transition model—a simulator—to generate imaginary data for training. Recent research has expanded this to bi-directional rollouts, where an agent learns both a forward model (predicting the next state) and a backward model (predicting the previous state).

However, conventional bi-directional methods typically aggregate all imaginary data into a single RL objective. This approach overlooks a critical insight: backward trajectories originating from high-reward states are inherently more informative than random exploration. They represent successful paths that the agent should prioritize learning. BIFRL leverages this by reframing backward rollouts as a source for imitation learning, effectively teaching the agent how to "reach" high-value regions of the state space.

The BIFRL Framework

The BIFRL architecture integrates bi-directional dynamics with a novel "Reach and Explore" training strategy. The framework consists of four primary components:

1. Bi-directional Dynamics and Backward Policy

The agent maintains two models: a forward dynamics model $p(s_{t+1}|s_t, a_t)$ and a backward dynamics model $p(s_{t-1}|s_t, a_{t-1})$. To facilitate backward rollouts, the agent also learns a specific backward policy $\tilde{\pi}(a_{t-1}|s_t)$, which predicts the action most likely to have resulted in the current state.

2. Value-Regularized GAN (VGAN)

In the early stages of training, high-value states are rare. To mitigate this sparsity, BIFRL employs a Value-Regularized Generative Adversarial Network (VGAN). Unlike standard GANs, the VGAN generator is optimized not only for realism but also to maximize the predicted Q-value of the generated states. This allows the system to "hallucinate" high-value starting points for rollouts even before the agent has naturally discovered them in the environment.

3. The "Reach and Explore" Strategy

BIFRL bifurcates the learning process based on the direction of the rollout:

  • Backward Imitation (Reach): Starting from high-value states (either real or hallucinated), the agent rolls backward in time. These traces are treated as expert demonstrations. The agent utilizes behavior cloning (Imitation Learning) to learn the policy required to reach these desirable states.
  • Forward Reinforcement (Explore): From those same high-value states, the agent performs forward rollouts using its current policy to explore potential future rewards. These trajectories are optimized using standard Soft Actor-Critic (SAC) objectives.

Theoretical Considerations: The $k_b < k_f$ Rule

A key contribution of the BIFRL research is the analysis of error accumulation. The authors demonstrate that "policy divergence"—the discrepancy between the learned backward policy and the true inverse of the expert behavior—accumulates more rapidly than standard model error. Consequently, they establish that backward rollouts ($k_b$) should generally be shorter than forward rollouts ($k_f$) to maintain training stability and prevent the agent from learning from highly inaccurate "expert" traces.

Experimental Results

The framework was evaluated on several MuJoCo physics engine benchmarks, including Hopper, Walker2D, and Ant, as well as "Non-Termination" (NT) variants where episodes do not end upon failure.

Sample Efficiency and Performance

BIFRL demonstrated superior sample efficiency compared to state-of-the-art baselines like MBPO and BMPO. On the Ant and Hopper tasks, BIFRL reached peak performance levels with significantly fewer environment steps. Furthermore, BIFRL often matched or exceeded the asymptotic performance of model-free SAC while requiring only a fraction of the total data.

Ablation Studies

Ablation experiments confirmed the necessity of both the VGAN and the Imitation Learning components. Removing the VGAN led to slower convergence due to the lack of high-value starting states, while removing the imitation objective (treating backward data as standard RL data) resulted in a marked decrease in learning speed, validating the "backward traces as experts" hypothesis.

Conclusion and Future Work

BIFRL provides a robust method for utilizing bi-directional models by distinguishing between the roles of forward and backward data. By treating the path to success as a demonstration to be imitated, the framework bridges the gap between imitation learning and reinforcement learning.

While highly effective, the framework's reliance on the critic (Q-value network) means that early-stage value overestimation could lead the VGAN to generate "delusional" states. Future research will focus on improving the robustness of the VGAN and applying BIFRL to more complex, high-dimensional environments where high-value states are exceptionally sparse.

Summary updated

BibTeX
@inproceedings{pan2022backward,
  author = {Yuxin Pan and Fangzhen Lin},
  title = {{Backward Imitation and Forward Reinforcement Learning via Bi-directional Model Rollouts}},
  booktitle = {IEEE/RSJ International Conference on Intelligent Robots and Systems},
  year = {2022},
  doi = {10.1109/iros47612.2022.9981869},
}
Figure