Abstract: While using shaped rewards can be beneficial when solving sparse reward tasks, their successful application often requires careful engineering and is problem specific. For instance, in tasks where the agent must achieve some goal state, simple distance-to-goal reward shaping often fails, as it renders learning vulnerable to local optima. We introduce a simple and effective model-free method to learn from shaped distance-to-goal rewards on tasks where success depends on reaching a goal state. Our method introduces an auxiliary distance-based reward based on pairs of rollouts to encourage diverse exploration. This approach effectively prevents learning dynamics from stabilizing around local optima induced by the naive distance-to-goal reward shaping and enables policies to efficiently solve sparse reward tasks. Our augmented objective does not require any additional reward engineering or domain expertise to implement and converges to the original sparse objective as the agent learns to solve the task. We demonstrate that our method successfully solves a variety of hard-exploration tasks (including maze navigation and 3D construction in a Minecraft environment), where naive distance-based reward shaping otherwise fails, and intrinsic curiosity and reward relabeling strategies exhibit poor performance.
Introduction
In Reinforcement Learning (RL), "sparse rewards" are the gold standard for defining tasks: an agent receives a signal only upon completion (e.g., finishing a maze). However, learning from these is incredibly slow because the agent spends most of its time wandering aimlessly. To accelerate learning, researchers often use reward shaping, adding a "warmer/colder" signal based on the distance to the goal. While intuitive, this often backfires. If a physical or logical barrier exists between the agent and the goal, a naive distance-based reward will push the agent into a local optimum—a state that is spatially "closer" but functionally a dead end.
Sibling Rivalry: A Model-Free Approach
The authors introduce Sibling Rivalry (SR), a method that balances exploration and exploitation using paired rollouts to "de-stabilize" these local optima.
The core mechanism involves:
- Sibling Rollouts: For every training step, the agent performs two independent "sibling" trajectories ($\tau^a$ and $\tau^b$) starting from the same initial state and aiming for the same goal.
- Mutual Relabeling: The trajectories are compared. The one that gets closer to the goal is the "closer sibling" ($\tau^c$), and the other is the "farther sibling" ($\tau^f$).
- The Anti-Goal: Each sibling treats the other’s terminal state as an "anti-goal." The reward function is modified so that the agent is rewarded for getting closer to the actual goal and for staying away from the region where its sibling got stuck.
- Selective Exclusion: To prevent the policy from converging too quickly to a sub-optimal "stable" region, the closer sibling is often excluded from the gradient update unless it reaches the goal or falls within a specific inclusion threshold ($\epsilon$).
- Self-Scheduling: As the agent learns to reach the goal consistently, the "anti-goal" signal naturally disappears, and the reward function converges back to the original sparse objective.
Experimental Evaluation
SR was evaluated across four environments designed to trigger local optima:
- 2D Point-Maze: Navigating a complex continuous maze.
- U-Shape Ant Maze: A high-dimensional MuJoCo environment where a quadruped must navigate a U-shaped corridor.
- 2D Discrete Pixel-Grid: A "bit-flipping" task where toggling bits to match a target can initially increase distance to the goal.
- 3D Construction in Minecraft: Building structures from a first-person perspective where placing blocks incorrectly is a significant local optimum.
Results and Discussion
In the 2D Point-Maze and Ant Maze, SR achieved a near 100% success rate, while standard PPO and Curiosity-based methods (ICM) almost never found the goal. In the Pixel-Grid task, SR bypassed the "toggle trap" that causes naive agents to avoid action entirely.
The paper identifies that local optima are essentially regions where the gradient of a naive shaped reward points toward a dead end. SR provides a functional reason to avoid these "stable" regions. Unlike hand-engineered penalties, the anti-goal in SR is dynamic, representing exactly where the policy is currently getting stuck.
While SR is model-free and doesn't require training auxiliary networks, it requires two rollouts for every update, effectively doubling the number of environment interactions per gradient step. Additionally, the inclusion threshold ($\epsilon$) is a critical hyperparameter; if set incorrectly, it can lead to over-exploration or failure to escape local optima. Future work may extend this "sibling" logic to off-policy architectures or multi-sibling groups.
Summary updated