Discovering Hierarchical Achievements in Reinforcement Learning via Contrastive Learning

Seungyong Moon, Junyoung Yeom, Bumsoo Park, Hyun Oh Song

TL;DR

Researchers have developed a novel method called Achievement Distillation that allows simple, model-free reinforcement learning agents to master complex, hierarchical tasks. By using contrastive learning to help agents predict their next "achievement," the method achieves state-of-the-art performance on the challenging Crafter benchmark while using only 4% of the parameters required by massive model-based competitors.

What Problem Does This Paper Address?

In open-world, procedurally generated environments (like Minecraft), agents face a "hierarchical achievement" problem. To get a diamond, you first need an iron pickaxe; to get that, you need a furnace and stone tools, and so on. This requires two difficult traits: long-term reasoning (planning many steps ahead) and generalization (applying these rules to a brand-new map every time).

Traditionally, researchers believed you needed complex "world models" (which imagine future states) or explicit "hierarchical planners" to solve this. However, these methods are often computationally expensive, require millions of interactions, or need prior knowledge of the achievement graph. The authors ask: Can we achieve the same results with a simpler, model-free approach?

Methods

The authors start with Proximal Policy Optimization (PPO), a standard model-free algorithm. They first show that simply applying modern implementation "best practices"—like larger networks, layer normalization, and value normalization—makes PPO surprisingly competitive.

Building on this, they introduce Achievement Distillation, a self-supervised auxiliary task integrated into PPO training. It consists of two main components:

  1. Intra-trajectory Achievement Prediction: The agent learns to associate its current state and action with the next achievement it is about to unlock within the same episode. This encourages the encoder to capture features relevant to future goals.
  2. Cross-trajectory Achievement Matching: Since the environment changes every episode but the "rules" (the achievement hierarchy) stay the same, the authors use Optimal Transport to match achievements across different episodes. This forces the agent to learn generalizable representations of achievements that aren't tied to a specific map layout.
Score curves showing how optimized PPO variants significantly outperform the default implementation.

Datasets

  • Crafter: The primary benchmark. It is a 2D, Minecraft-inspired survival game with 22 hierarchical achievements. It is procedurally generated and partially observable, making it a rigorous test for RL agents.
  • Procgen Heist: A benchmark focused on generalization where an agent must collect keys in a specific order to reach a gem.
  • MiniGrid: A grid-world environment used to test the method's effectiveness on deeper achievement graphs.
Overview of the Crafter environment and its hierarchical achievement graph.

Key Insights and Ideas

  • PPO is a "Sleeping Giant": With the right architecture and normalization, PPO outperforms the previously state-of-the-art DreamerV3 on Crafter.
  • Latent Foresight: The authors discovered that even a standard PPO agent's internal representations contain some information about the next achievement, but the confidence is low. Achievement Distillation acts as a "magnifying glass" for this latent knowledge.
  • Optimal Transport for Alignment: Using Optimal Transport allows the agent to discover which events in one run correspond to events in another run without needing manual labels, which is crucial for scaling to open-ended environments.
Figure 3: Histogram for the confidence of next achievement prediction.

Key Results

  • State-of-the-Art Performance: On Crafter, the proposed method achieved a score of 21.79%, significantly higher than the previous best, DreamerV3 (14.77%).
  • Efficiency: The model uses only 9 million parameters, which is roughly 4% of the 201 million parameters used by DreamerV3.
  • Challenging Tasks: The agent successfully learned to collect iron (a 20x improvement over DreamerV3) and even managed to collect diamonds—the most difficult task in the game—on individual runs.
  • Versatility: The method also improved performance when applied to value-based algorithms (like QR-DQN) and different environments (Heist and MiniGrid).
Score curves showing how optimized PPO variants significantly outperform the default implementation.

Limitations and Future Work

The main limitation is that the method currently relies on the agent receiving a reward signal when an achievement is unlocked. While this is a "sparse" reward, it still provides a hint. The authors suggest that a promising future direction is developing a version of this method that can distinguish and learn achievements in a fully unsupervised manner, perhaps by combining it with curiosity-driven exploration techniques.

Summary updated

Abstract

Discovering achievements with a hierarchical structure in procedurally generated environments presents a significant challenge. This requires an agent to possess a broad range of abilities, including generalization and long-term reasoning. Many prior methods have been built upon model-based or hierarchical approaches, with the belief that an explicit module for long-term planning would be advantageous for learning hierarchical dependencies. However, these methods demand an excessive number of environment interactions or large model sizes, limiting their practicality. In this work, we demonstrate that proximal policy optimization (PPO), a simple yet versatile model-free algorithm, outperforms previous methods when optimized with recent implementation practices. Moreover, we find that the PPO agent can predict the next achievement to be unlocked to some extent, albeit with limited confidence. Based on this observation, we introduce a novel contrastive learning method, called achievement distillation, which strengthens the agent's ability to predict the next achievement. Our method exhibits a strong capacity for discovering hierarchical achievements and shows state-of-the-art performance on the challenging Crafter environment in a sample-efficient manner while utilizing fewer model parameters.
BibTeX
@inproceedings{moon2023discovering,
  author = {Seungyong Moon and Junyoung Yeom and Bumsoo Park and Hyun Oh Song},
  title = {{Discovering Hierarchical Achievements in Reinforcement Learning via Contrastive Learning}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2023},
  url = {https://openreview.net/forum?id=cUuXVaMmmv},
}
Figure