SLOPE: Optimistic Potential Landscape Shaping for Model-based Reinforcement Learning

Yao-Hui Li, Zeyu Wang, Xin Li, Wei Pang, Yingfang Yuan, Zhengkun Chen, Boya Zhang, Riashat Islam, Alex Lamb, Yonggang ZhangarXiv 2026 · arXiv

TL;DR

SLOPE (Shaping Landscapes with Optimistic Potential Estimates) solves the "gradient starvation" problem in sparse-reward reinforcement learning by transforming binary success signals into dense, informative potential landscapes. By combining Potential-Based Reward Shaping with optimistic distributional value estimates, it provides the directional guidance necessary for effective planning even when rewards are extremely rare.

What Problem Does This Paper Address?

Model-Based Reinforcement Learning (MBRL) is highly valued for its sample efficiency, as it learns a "world model" to simulate future trajectories. However, it notoriously struggles in sparse reward settings—scenarios where a robot might only receive a signal of '1' upon total task completion and '0' otherwise.

In these cases, standard reward models often collapse, predicting zero everywhere. This creates a "flat" optimization landscape for planners like MPPI (Model Predictive Path Integral). Without a "slope" to follow, the planner has no way to distinguish between a trajectory that is moving toward the goal and one that is simply idling. While researchers have tried manual reward shaping or using expert demonstrations, these often require tedious task-specific engineering or fail to generalize.

Figure 1: Key challenges in sparse-reward MBRL.

Methods

The authors introduce SLOPE, a framework that fundamentally shifts reward modeling from predicting sparse scalars to constructing informative potential landscapes. It consists of three primary pillars:

  1. Model-Based Planning via Potential Landscapes: Instead of fitting the sparse ground-truth reward, SLOPE uses Potential-Based Reward Shaping (PBRS). It treats the agent’s own learned Q-value function as a "potential surface" over the state space. This transforms the flat reward structure into a dense landscape, providing step-by-step directional guidance during planning rollouts.
  2. Optimism-Driven Landscape Shaping: In sparse settings, standard value estimation often suffers from severe underestimation because success is so rare. SLOPE addresses this using Quantile-weighted Cross-Entropy (QCE). Instead of predicting the expected (mean) reward, it estimates a high-confidence upper bound. This "optimistic" bias amplifies rare success signals, ensuring the shaped landscape has sufficient gradients to drive exploration.
  3. Accelerated Training Strategies: SLOPE builds on the MoDem framework, using a three-phase training pipeline (pre-training, seeding, and interactive learning). It introduces enhancements like initializing the planner's sampling distribution from a prior policy and actively augmenting the demonstration buffer with successful trajectories encountered during training.
Figure 2: Framework of SLOPE. Building upon MoDem’s multi-phase accelerated learning frame- work, we also introduce two training enhancements: (i) initializing MPPI sampling distribution from the prior policy πPrior, and (ii) augmenting the demonstration buffer with successful trajectories. The shaped reward er is used for training both the reward model Rθ and the Q value function Qθ. The environment input consists of multimodal observations o = (x, q), where x denotes raw RGB images captured by the robot’s camera, and q represents the robot’s proprioceptive sensory information, e.g., the gripper state. Subsequent observations are encoded by the target encoder (illustrated in grey) which is updated via an exponential moving average (EMA) of the online encoder.

Datasets

The authors conducted an extensive evaluation across 30+ tasks and 5 major benchmarks:

  • ManiSkill3: 5 complex manipulation tasks.
  • Meta-World: 10 tasks ranging from medium to "very hard" difficulty.
  • Adroit: 2 high-dimensional dexterous manipulation tasks.
  • Robosuite: 3 robotic simulation tasks.
  • DeepMind Control Suite (DMC): 8 locomotion tasks used to test performance in dense-reward settings.
  • Real-World Deployment: Evaluated on a Realman RM75-B robotic arm for tasks like "Press Button," "Push Cube," and "Grasp Cube."

Key Insights and Ideas

  • From Scalars to Gradients: The core "aha moment" is that a planner doesn't just need to know if it succeeded; it needs a gradient showing how to succeed. By using the Q-function as a potential, the reward model is forced to encode local value gradients at every time step.
  • The Power of Optimism: In sparse environments, the "average" outcome is failure. If you train a model on the average, it learns to expect failure. By focusing on the upper quantiles (the "best-case" scenarios), SLOPE preserves the signal from rare successful trajectories that would otherwise be washed out.
  • Theoretical Soundness: The authors provide proofs showing that anchoring the potential to a slow-moving target network (via EMA) preserves the optimal policy of the original MDP and ensures the Bellman operator remains a strict contraction, guaranteeing convergence.

Key Results

SLOPE consistently outperformed leading baselines (like TD-MPC2, MoDem, and DEMO3) across all tested environments:

  • Simulation: On the ManiSkill3 benchmark, SLOPE achieved a 92.0% average success rate, significantly higher than the best baseline (DEMO3-semi-sparse at 74.0%) and far exceeding the vanilla TD-MPC2 (0%).
  • Real-World Robotics: In the challenging "Grasp Cube" task, SLOPE attained a 65.0% success rate with the lowest intervention rate (21.3%), while the model-free SAC baseline failed completely (0% success).
  • Generality: Even in dense reward tasks (DMC), SLOPE showed faster convergence than the backbone TD-MPC2, suggesting that potential-based shaping acts as an effective regularizer even when rewards aren't sparse.

Limitations and Future Work

The authors acknowledge a fundamental tension between optimistic exploration and safety. Because SLOPE aggressively scales potential to encourage exploration, it might inadvertently lead an agent into high-risk, unvisited regions of the state space.

To address this, the authors suggest that future work could extend the framework into Safety-Aware MBRL. This would involve incorporating safety-centric priors, such as Control Barrier Functions (CBF), to construct "repulsive" potential fields that steer the agent away from hazardous states while the optimistic potential pulls it toward the goal.

Summary updated

Abstract

Model-based reinforcement learning (MBRL) is sample-efficient but struggles in sparse reward settings. A critical bottleneck arises from the lack of informative gradients in sparse settings, where standard reward models often yield flat landscapes that struggle to guide planning. To address this challenge, we propose Shaping Landscapes with Optimistic Potential Estimates (SLOPE), a novel framework that shifts reward modeling from predicting sparse scalars to constructing informative potential landscapes. SLOPE employs optimistic distributional regression to estimate high-confidence upper bounds, which amplifies rare success signals and ensures sufficient exploration gradients. Evaluations on 30+ tasks across 5 benchmarks and real-world robotic deployments, demonstrate that SLOPE consistently outperforms leading baselines in fully sparse, semi-sparse, and dense rewards.
BibTeX
@misc{li2026slope,
  author = {Yao-Hui Li and Zeyu Wang and Xin Li and Wei Pang and Yingfang Yuan and Zhengkun Chen and Boya Zhang and Riashat Islam and Alex Lamb and Yonggang Zhang},
  title = {{SLOPE: Optimistic Potential Landscape Shaping for Model-based Reinforcement Learning}},
  year = {2026},
  eprint = {2602.03201},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2602.03201},
}
Figure