Reinforcement Learning (RL) agents often struggle with sparse reward tasks—scenarios where a reward is only granted upon final success, like reaching the center of a maze. To accelerate learning, practitioners frequently employ "reward shaping," providing dense, intermediate rewards such as the distance to the goal. However, this introduces a new problem: local optima.
The Problem with Distance-Based Rewards
Distance-based rewards are a double-edged sword. Consider an agent separated from its goal by a U-shaped wall. A naive distance-to-goal reward will pull the agent directly into the wall because it is physically "closer" to the target. To succeed, the agent must temporarily move away from the goal to navigate around the obstacle. Standard algorithms often get trapped in these local optima, and existing exploration strategies like intrinsic curiosity or Hindsight Experience Replay (HER) frequently fall short in these hard-exploration scenarios.
The Sibling Rivalry Method
To solve this, researchers introduced Sibling Rivalry (SR), a model-free RL technique that prevents agents from getting stuck by using self-balancing shaped rewards. The core mechanism involves sampling pairs of trajectories—"siblings"—from the exact same starting state.
- Identify the "Winner" and "Loser": For every start state, the policy runs twice, generating two sibling trajectories ($\tau^a$ and $\tau^b$). One sibling will inevitably terminate closer to the goal ($\tau^c$) than the other ($\tau^f$).
- Mutual Relabeling (The Anti-Goal): SR modifies the reward function by treating the terminal state of one sibling as an "anti-goal" for the other. If one sibling gets trapped in a local optimum (like the corner of a wall), the other is actively rewarded for avoiding that specific area while still pursuing the true goal.
- Selective Exclusion: To prevent the policy from settling for a "good enough" local optimum, the algorithm selectively filters training data. It consistently trains on the farther (worse) sibling to force exploration away from failure points, but only trains on the closer sibling if it actually reaches the goal or if both siblings are already very close to each other.
Key Insights and Dynamics
Unlike standard curiosity-based exploration that simply seeks "novel" states, SR specifically targets and penalizes states that frustrate the distance-to-goal metric. Because this "anti-goal" penalty is relative, it does not alter the global optimum—the highest reward is still achieved by reaching the actual goal.
Furthermore, SR acts as an automatic curriculum. Early in training, the sibling trajectories vary wildly, providing a strong exploratory push. As the agent masters the task and both siblings consistently reach the goal, the auxiliary anti-goal reward naturally vanishes, converging back to the original sparse objective.
Results and Performance
The authors evaluated SR across several challenging environments, including a 2D Point-Maze, a high-dimensional Ant-Maze, a discrete Pixel-Grid task, and a 3D Minecraft construction environment.
- Overcoming Local Optima: In the 2D Point-Maze, standard Proximal Policy Optimization (PPO) with naive distance rewards failed 100% of the time, getting permanently stuck. PPO augmented with SR solved the maze consistently.
- Outperforming Baselines: SR demonstrated superior performance compared to state-of-the-art exploration methods like Intrinsic Curiosity Modules (ICM) and HER across multiple tasks.
- Scalability: In the complex Minecraft construction task, naive distance rewards caused the agent to stop placing blocks entirely to avoid accidentally increasing its distance to the goal state. SR successfully overcame this, achieving high construction accuracy across thousands of unique target structures.
Limitations
While powerful, SR has limitations. It requires the ability to reset the environment to the exact same starting state for sibling rollouts, which is trivial in simulation but challenging in real-world robotics. Additionally, sampling two trajectories per update doubles the environment interactions per iteration, though this sample efficiency trade-off is often worth it to solve otherwise intractable tasks. Finally, the current formulation is designed primarily for on-policy algorithms, leaving its adaptation to off-policy methods as an area for future work.
Summary updated