TL;DR
GOPlan introduces a model-based framework for offline goal-conditioned reinforcement learning (GCRL) that overcomes the challenges of limited offline datasets and out-of-distribution (OOD) goals. By pairing an advantage-weighted Conditional GAN (to capture sharp, multi-modal action distributions) with an uncertainty-aware trajectory reanalysis step across both intra- and inter-trajectory goals, GOPlan generates reliable synthetic rollouts to iteratively improve policy performance.
What Problem Does This Paper Address?
In offline goal-conditioned reinforcement learning (GCRL), agents learn to reach diverse target states using only pre-collected datasets without active environment interaction. While model-based "reanalysis" techniques (imagining new trajectories using learned world models) have succeeded in single-task offline RL, porting them to multi-goal environments exposes two major hurdles:
- Multi-modal action distributions: In multi-goal datasets, the same state can legitimately lead to entirely different actions depending on the target goal. Traditional policies (e.g., Gaussians or standard VAEs) tend to interpolate between these distinct modes, generating out-of-distribution (OOD), physically nonsensical actions during multi-step planning.
- Goal selection for reanalysis: It is non-trivial to decide which goals to imagine trajectories for without straying outside the region of state-action space where the learned dynamics model is reliable.
Methods
GOPlan tackles these challenges via a two-stage algorithm:
[Offline Dataset] ──> (1) Pretrain Prior Policy (Advantage-Weighted CGAN) + Dynamics Ensemble
│
▼
(2) Model-Based Reanalysis (Intra- & Inter-Trajectory Goals)
│
▼ [Filter via Model Disagreement]
(3) Finetune Policy & Value Function on Reanalysis Buffer
- Pretraining Stage (Advantage-Weighted CGAN):
- The policy is modeled using an exponentially advantage-weighted Conditional GAN (CGAN). Unlike standard Gaussian or VAE-based policies that suffer from mode-averaging, the adversarial objective preserves distinct action modes while prioritizing high-reward behaviors.
- An ensemble of $N$ dynamics models is simultaneously trained to predict state residuals ($s_{t+1} - s_t$).
- Reanalysis Stage (Multi-Goal Planning & Filtering):
- Intra-trajectory goals: Reanalyzing states targeting future goals within the same recorded trajectory to find shorter, more optimal paths.
- Inter-trajectory goals: Stitching behaviors together by setting goals from entirely different trajectories.
-
Uncertainty quantification: Imagined rollouts are produced using a model-based planner. Rollout uncertainty is measured by dynamics model disagreement across the ensemble: $$U(\tau) = \max_{0 \le t < T} \frac{1}{N} \sum_{i=1}^N \|M_i(s_t, a_t) - \bar{s}_{t+1}\|_2^2$$ Trajectories exceeding an uncertainty threshold $u$ are filtered out to prevent policy exploitation of model errors.
-
High-quality imagined trajectories populate a reanalysis buffer ($B_{re}$) used to iteratively finetune the value function, discriminator, and policy.
Datasets
The authors evaluate GOPlan on two main suites of manipulation benchmarks:
- Standard Offline GCRL Tasks [27]:
FetchPush,FetchPick,FetchSlide, andHandReach. Tested under two data regimes: - Normal: $2 \times 10^6$ transitions.
- Small (data-budgeted): $2 \times 10^5$ transitions ($1/10\text{th}$ scale).
- Data collected using a DDPG policy with hindsight relabeling, corrupted with Gaussian noise ($\sigma = 0.2$) to enforce behavioral multi-modality.
- OOD Generalization Benchmarks [26]: Four task groups (
FetchPush Left-Right,FetchPush Near-Far,FetchPick Left-Right, andFetchPick Low-High) containing 5,000 transitions each. Evaluates the policy on unseen combinations of initial states and target goal configurations (e.g., training onRight2Righttrajectories but testing onRight2LeftorLeft2Right).
Key Insights and Ideas
- Avoiding mode-averaging via CGANs: Prior works often use Gaussian or CVAE policies. Due to Euclidean regularization on VAE decoders or Gaussian unimodality, they generate blended, invalid actions between valid trajectories. Advantage-weighted CGAN preserves sharp mode separation while weighting modes by their expected returns.
- Expanding dataset coverage via inter-trajectory stitching: While standard hindsight experience replay typically relabels goals within the same trajectory, GOPlan leverages learned dynamics to generate valid synthetic connections between different trajectories, significantly enhancing out-of-distribution goal reachability.
- Disagreement-gated data augmentation: Because model rollouts quickly compound errors in offline regimes, strict disagreement filtering ensures that only imagined data with high epistemic certainty enters the training buffer.
Key Results
- SOTA on Standard Benchmarks: GOPlan consistently matches or exceeds top baselines (BC, GCSL, WGCSL, GEAW, AM, CRL, and g-TD3-BC).
- On the standard
HandReachtask, GOPlan achieves 28.28 ± 5.3 (nearest competitor WGCSL: 26.73 ± 1.2; model-free baselines drop below 7.0). - Robustness to Low Data Budgets: In the $10\times$ smaller dataset setting, GOPlan maintains strong performance where other methods collapse:
- On small
HandReach, GOPlan achieves 10.11 ± 1.4, while the next-best baseline manages only 1.6 ± 2.3. - On small
FetchSlide, GOPlan achieves 5.04 ± 0.4, beating CRL (4.74) and WGCSL (0.22). - OOD Goal Generalization: In out-of-distribution evaluations, GOPlan (and its test-time planning variant, GOPlan2) achieved higher average returns and tighter lower-bound performance across all four OOD task distributions compared to prior methods.
Limitations and Future Work
- Computational Overhead: Iterative model-based trajectory generation and test-time planning (in GOPlan2) introduce non-trivial computational complexity compared to simple supervised or model-free approaches.
- State-Based Representation: Experiments are focused on state-residual prediction in continuous robotic control; scaling this approach to high-dimensional pixel observations (e.g., latent dynamics) is not explored.
- Heuristic Uncertainty Thresholds: Filtering invalid rollouts relies on a scalar disagreement threshold $u$, which may require task-specific tuning across diverse environment dynamics.
Summary updated