Scaling Laws for Imitation Learning in NetHack

Jens Tuyls, Dhruv Madeka, Kari Torkkola, Dean P. Foster, Karthik Narasimhan, Sham M. Kakade

Abstract

While scaling laws have revolutionized natural language processing, their applicability to complex decision-making environments remains less explored. This paper investigates scaling laws for Imitation Learning (IL) in single-agent games, specifically focusing on the challenging roguelike NetHack. We demonstrate that both validation loss and environment return follow predictable power laws relative to compute, model size, and data volume. By leveraging these laws to optimize model configurations, we achieve a new state-of-the-art score in NetHack, outperforming previous offline agents by 1.7x.

Introduction: Is Scale All You Need for Games?

Imitation Learning (IL), particularly Behavioral Cloning (BC), is a fundamental technique for training agents to mimic expert behavior. Despite its utility, IL often struggles to fully replicate expert performance, a failure frequently attributed to algorithmic issues like causal confusion. While the Large Language Model (LLM) community has found that increasing scale predictably improves performance, the Reinforcement Learning (RL) and IL communities have traditionally focused on algorithmic refinements. This research investigates whether scaling model size and data volume provides similar predictable benefits for agents in complex game environments.

Methods

The study focuses on Behavioral Cloning (BC) within the NetHack Learning Environment. To characterize scaling behavior, the authors employ two main techniques:

  1. isoFLOP Profiles: Agents are trained across multiple orders of magnitude of compute (FLOPs). For each compute budget, various model sizes are tested to identify the "optimal" configuration—the specific balance of parameters and dataset size that minimizes validation loss.
  2. Parametric Fitting: The relationship between loss ($L$), parameters ($N$), and data ($D$) is modeled using functional forms inspired by the Cobb-Douglas production function, allowing for the prediction of performance at scales beyond those currently tested.
Dev loss vs. Parameters for various compute budgets (FLOPs). Each curve represents a fixed compute budget, showing an optimal model size that minimizes loss.

Datasets

The primary experimental testbed is NetHack, a procedurally generated roguelike known for its extreme difficulty and partial observability. The researchers utilized the NLD-AA dataset and introduced NLD-AA-L, a massive dataset containing 55 billion samples generated by the high-performing rule-based expert AutoAscend. Baseline experiments were also conducted on eight Atari 2600 games using PPO-generated expert data.

Key Insights

A critical finding is the strong correlation between cross-entropy loss (the training objective) and environment return (actual game performance). Both metrics scale smoothly with compute, enabling researchers to use relatively inexpensive loss curves to forecast the compute requirements needed to achieve specific game scores.

Validation loss follows a predictable power law as a function of total training compute (FLOPs).

However, unlike Atari agents which eventually reach expert-level performance, NetHack agents exhibit a performance plateau. This is likely due to partial observability; the agent lacks the same information parity as the expert (e.g., historical state or inventory tracking), suggesting that while scale improves performance, it cannot entirely compensate for missing information.

Key Results

By applying these scaling laws, the authors trained a "Scaled-BC" agent that achieved a score of 7784 on the Human Monk role in NetHack. This represents a 1.7x improvement over the previous state-of-the-art offline agent (4504). The empirical results across all tested Atari games and NetHack showed that optimal loss and return scaled smoothly with the compute budget, and actual results matched theoretical predictions almost perfectly.

Discussion and Future Work

The results suggest that many perceived failures of imitation learning in literature may stem from insufficient scale rather than fundamental algorithmic flaws. Future work should explore how these scaling laws interact with memory-intensive architectures to address the partial observability plateau in NetHack. Additionally, investigating the impact of expert data quality on the slope of the scaling law remains a key area for further research.

Summary updated

BibTeX
@misc{tuyls2023scaling,
  author = {Jens Tuyls and Dhruv Madeka and Kari Torkkola and Dean P. Foster and Karthik Narasimhan and Sham M. Kakade},
  title = {{Scaling Laws for Imitation Learning in NetHack}},
  year = {2023},
  eprint = {2307.09423},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2307.09423},
}
Figure