GOPlan: Goal-conditioned Offline Reinforcement Learning by Planning with Learned Models

Mianchu Wang, Rui Yang, Xi Chen, Meng Fang

TL;DR

GOPlan introduces a model-based framework for offline goal-conditioned reinforcement learning (GCRL) that overcomes the challenges of limited offline datasets and out-of-distribution (OOD) goals. By pairing an advantage-weighted Conditional GAN (to capture sharp, multi-modal action distributions) with an uncertainty-aware trajectory reanalysis step across both intra- and inter-trajectory goals, GOPlan generates reliable synthetic rollouts to iteratively improve policy performance.

What Problem Does This Paper Address?

In offline goal-conditioned reinforcement learning (GCRL), agents learn to reach diverse target states using only pre-collected datasets without active environment interaction. While model-based "reanalysis" techniques (imagining new trajectories using learned world models) have succeeded in single-task offline RL, porting them to multi-goal environments exposes two major hurdles:

  1. Multi-modal action distributions: In multi-goal datasets, the same state can legitimately lead to entirely different actions depending on the target goal. Traditional policies (e.g., Gaussians or standard VAEs) tend to interpolate between these distinct modes, generating out-of-distribution (OOD), physically nonsensical actions during multi-step planning.
  2. Goal selection for reanalysis: It is non-trivial to decide which goals to imagine trajectories for without straying outside the region of state-action space where the learned dynamics model is reliable.

Methods

GOPlan tackles these challenges via a two-stage algorithm:

[Offline Dataset] ──> (1) Pretrain Prior Policy (Advantage-Weighted CGAN) + Dynamics Ensemble
                             │
                             ▼
                      (2) Model-Based Reanalysis (Intra- & Inter-Trajectory Goals)
                             │
                             ▼ [Filter via Model Disagreement]
                      (3) Finetune Policy & Value Function on Reanalysis Buffer
  1. Pretraining Stage (Advantage-Weighted CGAN):
  2. The policy is modeled using an exponentially advantage-weighted Conditional GAN (CGAN). Unlike standard Gaussian or VAE-based policies that suffer from mode-averaging, the adversarial objective preserves distinct action modes while prioritizing high-reward behaviors.
  3. An ensemble of $N$ dynamics models is simultaneously trained to predict state residuals ($s_{t+1} - s_t$).
  4. Reanalysis Stage (Multi-Goal Planning & Filtering):
  5. Intra-trajectory goals: Reanalyzing states targeting future goals within the same recorded trajectory to find shorter, more optimal paths.
  6. Inter-trajectory goals: Stitching behaviors together by setting goals from entirely different trajectories.
  7. Uncertainty quantification: Imagined rollouts are produced using a model-based planner. Rollout uncertainty is measured by dynamics model disagreement across the ensemble: $$U(\tau) = \max_{0 \le t < T} \frac{1}{N} \sum_{i=1}^N \|M_i(s_t, a_t) - \bar{s}_{t+1}\|_2^2$$ Trajectories exceeding an uncertainty threshold $u$ are filtered out to prevent policy exploitation of model errors.

  8. High-quality imagined trajectories populate a reanalysis buffer ($B_{re}$) used to iteratively finetune the value function, discriminator, and policy.

Datasets

The authors evaluate GOPlan on two main suites of manipulation benchmarks:

  • Standard Offline GCRL Tasks [27]: FetchPush, FetchPick, FetchSlide, and HandReach. Tested under two data regimes:
  • Normal: $2 \times 10^6$ transitions.
  • Small (data-budgeted): $2 \times 10^5$ transitions ($1/10\text{th}$ scale).
  • Data collected using a DDPG policy with hindsight relabeling, corrupted with Gaussian noise ($\sigma = 0.2$) to enforce behavioral multi-modality.
  • OOD Generalization Benchmarks [26]: Four task groups (FetchPush Left-Right, FetchPush Near-Far, FetchPick Left-Right, and FetchPick Low-High) containing 5,000 transitions each. Evaluates the policy on unseen combinations of initial states and target goal configurations (e.g., training on Right2Right trajectories but testing on Right2Left or Left2Right).

Key Insights and Ideas

  • Avoiding mode-averaging via CGANs: Prior works often use Gaussian or CVAE policies. Due to Euclidean regularization on VAE decoders or Gaussian unimodality, they generate blended, invalid actions between valid trajectories. Advantage-weighted CGAN preserves sharp mode separation while weighting modes by their expected returns.
  • Expanding dataset coverage via inter-trajectory stitching: While standard hindsight experience replay typically relabels goals within the same trajectory, GOPlan leverages learned dynamics to generate valid synthetic connections between different trajectories, significantly enhancing out-of-distribution goal reachability.
  • Disagreement-gated data augmentation: Because model rollouts quickly compound errors in offline regimes, strict disagreement filtering ensures that only imagined data with high epistemic certainty enters the training buffer.

Key Results

  • SOTA on Standard Benchmarks: GOPlan consistently matches or exceeds top baselines (BC, GCSL, WGCSL, GEAW, AM, CRL, and g-TD3-BC).
  • On the standard HandReach task, GOPlan achieves 28.28 ± 5.3 (nearest competitor WGCSL: 26.73 ± 1.2; model-free baselines drop below 7.0).
  • Robustness to Low Data Budgets: In the $10\times$ smaller dataset setting, GOPlan maintains strong performance where other methods collapse:
  • On small HandReach, GOPlan achieves 10.11 ± 1.4, while the next-best baseline manages only 1.6 ± 2.3.
  • On small FetchSlide, GOPlan achieves 5.04 ± 0.4, beating CRL (4.74) and WGCSL (0.22).
  • OOD Goal Generalization: In out-of-distribution evaluations, GOPlan (and its test-time planning variant, GOPlan2) achieved higher average returns and tighter lower-bound performance across all four OOD task distributions compared to prior methods.

Limitations and Future Work

  • Computational Overhead: Iterative model-based trajectory generation and test-time planning (in GOPlan2) introduce non-trivial computational complexity compared to simple supervised or model-free approaches.
  • State-Based Representation: Experiments are focused on state-residual prediction in continuous robotic control; scaling this approach to high-dimensional pixel observations (e.g., latent dynamics) is not explored.
  • Heuristic Uncertainty Thresholds: Filtering invalid rollouts relies on a scalar disagreement threshold $u$, which may require task-specific tuning across diverse environment dynamics.

Summary updated

Abstract

Offline goal-conditioned RL (GCRL) offers a feasible paradigm to learn general-purpose policies from diverse and multi-task offline datasets. Despite notable recent progress, the predominant offline GCRL methods have been restricted to model-free approaches, constraining their capacity to tackle limited data budgets and unseen goal generalization. In this work, we propose a novel two-stage model-based framework, Goal-conditioned Offline Planning (GOPlan), including (1) pretraining a prior policy capable of capturing multi-modal action distribution within the multi-goal dataset; (2) employing the reanalysis method with planning to generate imagined trajectories for funetuning policies. Specifically, the prior policy is based on an advantage-weighted Conditioned Generative Adversarial Networks that exhibits distinct mode separation to overcome the pitfalls of out-of-distribution (OOD) actions. For further policy optimization, the reanalysis method generates high-quality imaginary data by planning with learned models for both intra-trajectory and inter-trajectory goals. Through experimental evaluations, we demonstrate that GOPlan achieves state-of-the-art performance on various offline multi-goal manipulation tasks. Moreover, our results highlight the superior ability of GOPlan to handle small data budgets and generalize to OOD goals.
BibTeX
@article{wang2024goplan,
  author = {Mianchu Wang and Rui Yang and Xi Chen and Meng Fang},
  title = {{GOPlan: Goal-conditioned Offline Reinforcement Learning by Planning with Learned Models}},
  journal = {Accepted by TMLR},
  year = {2024},
  url = {https://openreview.net/forum?id=zOKAmm8R9B},
}
Figure