Introduction
The "holy grail" of robotics is an autonomous agent capable of learning a broad set of skills without constant human intervention. Traditionally, Reinforcement Learning (RL) has relied on hand-engineered reward functions for every specific task—such as providing a point for touching a door handle. This approach is labor-intensive and fails to scale to the complexity of the real world.
Self-supervised RL offers a solution by allowing agents to set their own goals. However, these agents often suffer from a "rich-get-richer" problem: they tend to sample goals from states they have already visited frequently. If a robot spends most of its time in the center of a room, it will continue to set goals in that area, failing to explore corners or interact with novel objects. Skew-Fit addresses this fundamental challenge of state coverage, enabling agents to systematically explore environments and learn to reach all possible states, even without prior knowledge of the environment's boundaries.
The Skew-Fit Algorithm
Skew-Fit is designed to maximize the entropy (diversity) of an agent's goal distribution. The authors formalize this as maximizing the mutual information $I(S; G)$ between states ($S$) and goals ($G$). This objective encourages the agent to both improve its ability to reach specific goals and to make those goals as diverse as possible.
The "Skew" Mechanism
The core of the algorithm is a simple yet mathematically grounded reweighting trick. The agent uses a Variational Autoencoder (VAE) to model the distribution of states it has encountered. To select new goals, the agent does not sample directly from the VAE. Instead, it reweights the samples using a "skewing" function:
$$w(s) \propto p(s)^\alpha$$
where $\alpha$ is a negative hyperparameter (typically -1). Because $\alpha$ is negative, states that have been visited less frequently receive higher weights.
Iterative Refinement
As the agent practices reaching these rare "skewed" goals, it begins to visit previously unexplored areas. The VAE is then updated with these new experiences, the once-rare states become common, and the agent's focus naturally shifts to the next frontier of unexplored territory. This iterative process causes the goal distribution to converge toward a uniform distribution over all valid states. Skew-Fit is built upon Reinforcement Learning with Imagined Goals (RIG), allowing it to handle high-dimensional image inputs by mapping them into a compact latent space.
Experimental Evaluation and Results
The researchers evaluated Skew-Fit across several challenging benchmarks, including 2D navigation, a simulated "Ant Labyrinth," and vision-based tasks like object pickup and puck pushing.
Key Insights
- Goal-Directed Exploration: Unlike curiosity-based methods that push an agent toward novelty through random wandering, Skew-Fit trains a policy to explicitly reach novel states, resulting in a reusable library of skills.
- Superior Coverage: In the Ant Labyrinth, Skew-Fit was the only method capable of exploring the entire maze, while prior methods like Hindsight Experience Replay (HER) remained clumped near the starting position.
Real-World Robotics Success
The most striking result was demonstrated on a Rethink Sawyer robot arm. Tasked with learning to open a door from scratch using only raw pixels from a Kinect camera, the robot achieved a 95% success rate after only 5.5 hours of interaction.
Crucially, the robot performed this task without a single human-defined reward; it simply "played" with the door until it had mastered the entire state space.
Conclusion and Future Work
Skew-Fit provides a powerful bridge between heuristic exploration and formal information theory. By simply reweighting the goal distribution to favor the rare, agents can autonomously discover complex behaviors in both simulation and the real world. While the method currently relies on the quality of the VAE's latent space and assumes finite-horizon tasks, it represents a significant step toward truly autonomous, "always-on" robotic learning.
Summary updated