Skew-Fit: State-Covering Self-Supervised Reinforcement Learning

Vitchyr H. Pong, Murtaza Dalal, Steven Lin, Ashvin Nair, Shikhar Bahl, Sergey Levine

Introduction

The "holy grail" of robotics is an autonomous agent capable of learning a broad set of skills without constant human intervention. Traditionally, Reinforcement Learning (RL) has relied on hand-engineered reward functions for every specific task—such as providing a point for touching a door handle. This approach is labor-intensive and fails to scale to the complexity of the real world.

Self-supervised RL offers a solution by allowing agents to set their own goals. However, these agents often suffer from a "rich-get-richer" problem: they tend to sample goals from states they have already visited frequently. If a robot spends most of its time in the center of a room, it will continue to set goals in that area, failing to explore corners or interact with novel objects. Skew-Fit addresses this fundamental challenge of state coverage, enabling agents to systematically explore environments and learn to reach all possible states, even without prior knowledge of the environment's boundaries.

The Skew-Fit Algorithm

Skew-Fit is designed to maximize the entropy (diversity) of an agent's goal distribution. The authors formalize this as maximizing the mutual information $I(S; G)$ between states ($S$) and goals ($G$). This objective encourages the agent to both improve its ability to reach specific goals and to make those goals as diverse as possible.

The "Skew" Mechanism

The core of the algorithm is a simple yet mathematically grounded reweighting trick. The agent uses a Variational Autoencoder (VAE) to model the distribution of states it has encountered. To select new goals, the agent does not sample directly from the VAE. Instead, it reweights the samples using a "skewing" function:

$$w(s) \propto p(s)^\alpha$$

where $\alpha$ is a negative hyperparameter (typically -1). Because $\alpha$ is negative, states that have been visited less frequently receive higher weights.

The Skew-Fit iterative process: by skewing the distribution of visited states to favor rare ones (indicated by larger circles), the agent proposes more diverse goals, eventually leading to uniform state coverage.

Iterative Refinement

As the agent practices reaching these rare "skewed" goals, it begins to visit previously unexplored areas. The VAE is then updated with these new experiences, the once-rare states become common, and the agent's focus naturally shifts to the next frontier of unexplored territory. This iterative process causes the goal distribution to converge toward a uniform distribution over all valid states. Skew-Fit is built upon Reinforcement Learning with Imagined Goals (RIG), allowing it to handle high-dimensional image inputs by mapping them into a compact latent space.

Experimental Evaluation and Results

The researchers evaluated Skew-Fit across several challenging benchmarks, including 2D navigation, a simulated "Ant Labyrinth," and vision-based tasks like object pickup and puck pushing.

Key Insights

  • Goal-Directed Exploration: Unlike curiosity-based methods that push an agent toward novelty through random wandering, Skew-Fit trains a policy to explicitly reach novel states, resulting in a reusable library of skills.
  • Superior Coverage: In the Ant Labyrinth, Skew-Fit was the only method capable of exploring the entire maze, while prior methods like Hindsight Experience Replay (HER) remained clumped near the starting position.

Real-World Robotics Success

The most striking result was demonstrated on a Rethink Sawyer robot arm. Tasked with learning to open a door from scratch using only raw pixels from a Kinect camera, the robot achieved a 95% success rate after only 5.5 hours of interaction.

A Rethink Sawyer robot learning to open a door to various angles from raw pixels. Skew-Fit allows the robot to master the task autonomously without any human-defined rewards.

Crucially, the robot performed this task without a single human-defined reward; it simply "played" with the door until it had mastered the entire state space.

Conclusion and Future Work

Skew-Fit provides a powerful bridge between heuristic exploration and formal information theory. By simply reweighting the goal distribution to favor the rare, agents can autonomously discover complex behaviors in both simulation and the real world. While the method currently relies on the quality of the VAE's latent space and assumes finite-horizon tasks, it represents a significant step toward truly autonomous, "always-on" robotic learning.

Summary updated

Abstract

Autonomous agents that must exhibit flexible and broad capabilities will need to be equipped with large repertoires of skills. Defining each skill with a manually-designed reward function limits this repertoire and imposes a manual engineering burden. Self-supervised agents that set their own goals can automate this process, but designing appropriate goal setting objectives can be difficult, and often involves heuristic design decisions. In this paper, we propose a formal exploration objective for goal-reaching policies that maximizes state coverage. We show that this objective is equivalent to maximizing goal reaching performance together with the entropy of the goal distribution, where goals correspond to full state observations. To instantiate this principle, we present an algorithm called Skew-Fit for learning a maximum-entropy goal distributions. We prove that, under regularity conditions, Skew-Fit converges to a uniform distribution over the set of valid states, even when we do not know this set beforehand. Our experiments show that combining Skew-Fit for learning goal distributions with existing goal-reaching methods outperforms a variety of prior methods on open-sourced visual goal-reaching tasks. Moreover, we demonstrate that Skew-Fit enables a real-world robot to learn to open a door, entirely from scratch, from pixels, and without any manually-designed reward function.
BibTeX
@inproceedings{pong2020skew,
  author = {Vitchyr H. Pong and Murtaza Dalal and Steven Lin and Ashvin Nair and Shikhar Bahl and Sergey Levine},
  title = {{Skew-Fit: State-Covering Self-Supervised Reinforcement Learning}},
  booktitle = {International Conference on Machine Learning},
  year = {2020},
  url = {https://openreview.net/forum?id=LYZDMYwlHhS},
}
Figure