TL;DR
Researchers have developed a novel method called Achievement Distillation that allows simple, model-free reinforcement learning agents to master complex, hierarchical tasks. By using contrastive learning to help agents predict their next "achievement," the method achieves state-of-the-art performance on the challenging Crafter benchmark while using only 4% of the parameters required by massive model-based competitors.
What Problem Does This Paper Address?
In open-world, procedurally generated environments (like Minecraft), agents face a "hierarchical achievement" problem. To get a diamond, you first need an iron pickaxe; to get that, you need a furnace and stone tools, and so on. This requires two difficult traits: long-term reasoning (planning many steps ahead) and generalization (applying these rules to a brand-new map every time).
Traditionally, researchers believed you needed complex "world models" (which imagine future states) or explicit "hierarchical planners" to solve this. However, these methods are often computationally expensive, require millions of interactions, or need prior knowledge of the achievement graph. The authors ask: Can we achieve the same results with a simpler, model-free approach?
Methods
The authors start with Proximal Policy Optimization (PPO), a standard model-free algorithm. They first show that simply applying modern implementation "best practices"—like larger networks, layer normalization, and value normalization—makes PPO surprisingly competitive.
Building on this, they introduce Achievement Distillation, a self-supervised auxiliary task integrated into PPO training. It consists of two main components:
- Intra-trajectory Achievement Prediction: The agent learns to associate its current state and action with the next achievement it is about to unlock within the same episode. This encourages the encoder to capture features relevant to future goals.
- Cross-trajectory Achievement Matching: Since the environment changes every episode but the "rules" (the achievement hierarchy) stay the same, the authors use Optimal Transport to match achievements across different episodes. This forces the agent to learn generalizable representations of achievements that aren't tied to a specific map layout.
Datasets
- Crafter: The primary benchmark. It is a 2D, Minecraft-inspired survival game with 22 hierarchical achievements. It is procedurally generated and partially observable, making it a rigorous test for RL agents.
- Procgen Heist: A benchmark focused on generalization where an agent must collect keys in a specific order to reach a gem.
- MiniGrid: A grid-world environment used to test the method's effectiveness on deeper achievement graphs.
Key Insights and Ideas
- PPO is a "Sleeping Giant": With the right architecture and normalization, PPO outperforms the previously state-of-the-art DreamerV3 on Crafter.
- Latent Foresight: The authors discovered that even a standard PPO agent's internal representations contain some information about the next achievement, but the confidence is low. Achievement Distillation acts as a "magnifying glass" for this latent knowledge.
- Optimal Transport for Alignment: Using Optimal Transport allows the agent to discover which events in one run correspond to events in another run without needing manual labels, which is crucial for scaling to open-ended environments.
Key Results
- State-of-the-Art Performance: On Crafter, the proposed method achieved a score of 21.79%, significantly higher than the previous best, DreamerV3 (14.77%).
- Efficiency: The model uses only 9 million parameters, which is roughly 4% of the 201 million parameters used by DreamerV3.
- Challenging Tasks: The agent successfully learned to collect iron (a 20x improvement over DreamerV3) and even managed to collect diamonds—the most difficult task in the game—on individual runs.
- Versatility: The method also improved performance when applied to value-based algorithms (like QR-DQN) and different environments (Heist and MiniGrid).
Limitations and Future Work
The main limitation is that the method currently relies on the agent receiving a reward signal when an achievement is unlocked. While this is a "sparse" reward, it still provides a hint. The authors suggest that a promising future direction is developing a version of this method that can distinguish and learn achievements in a fully unsupervised manner, perhaps by combining it with curiosity-driven exploration techniques.
Summary updated