Abstract
While scaling laws have revolutionized natural language processing, their applicability to complex decision-making environments remains less explored. This paper investigates scaling laws for Imitation Learning (IL) in single-agent games, specifically focusing on the challenging roguelike NetHack. We demonstrate that both validation loss and environment return follow predictable power laws relative to compute, model size, and data volume. By leveraging these laws to optimize model configurations, we achieve a new state-of-the-art score in NetHack, outperforming previous offline agents by 1.7x.
Introduction: Is Scale All You Need for Games?
Imitation Learning (IL), particularly Behavioral Cloning (BC), is a fundamental technique for training agents to mimic expert behavior. Despite its utility, IL often struggles to fully replicate expert performance, a failure frequently attributed to algorithmic issues like causal confusion. While the Large Language Model (LLM) community has found that increasing scale predictably improves performance, the Reinforcement Learning (RL) and IL communities have traditionally focused on algorithmic refinements. This research investigates whether scaling model size and data volume provides similar predictable benefits for agents in complex game environments.
Methods
The study focuses on Behavioral Cloning (BC) within the NetHack Learning Environment. To characterize scaling behavior, the authors employ two main techniques:
- isoFLOP Profiles: Agents are trained across multiple orders of magnitude of compute (FLOPs). For each compute budget, various model sizes are tested to identify the "optimal" configuration—the specific balance of parameters and dataset size that minimizes validation loss.
- Parametric Fitting: The relationship between loss ($L$), parameters ($N$), and data ($D$) is modeled using functional forms inspired by the Cobb-Douglas production function, allowing for the prediction of performance at scales beyond those currently tested.
Datasets
The primary experimental testbed is NetHack, a procedurally generated roguelike known for its extreme difficulty and partial observability. The researchers utilized the NLD-AA dataset and introduced NLD-AA-L, a massive dataset containing 55 billion samples generated by the high-performing rule-based expert AutoAscend. Baseline experiments were also conducted on eight Atari 2600 games using PPO-generated expert data.
Key Insights
A critical finding is the strong correlation between cross-entropy loss (the training objective) and environment return (actual game performance). Both metrics scale smoothly with compute, enabling researchers to use relatively inexpensive loss curves to forecast the compute requirements needed to achieve specific game scores.
However, unlike Atari agents which eventually reach expert-level performance, NetHack agents exhibit a performance plateau. This is likely due to partial observability; the agent lacks the same information parity as the expert (e.g., historical state or inventory tracking), suggesting that while scale improves performance, it cannot entirely compensate for missing information.
Key Results
By applying these scaling laws, the authors trained a "Scaled-BC" agent that achieved a score of 7784 on the Human Monk role in NetHack. This represents a 1.7x improvement over the previous state-of-the-art offline agent (4504). The empirical results across all tested Atari games and NetHack showed that optimal loss and return scaled smoothly with the compute budget, and actual results matched theoretical predictions almost perfectly.
Discussion and Future Work
The results suggest that many perceived failures of imitation learning in literature may stem from insufficient scale rather than fundamental algorithmic flaws. Future work should explore how these scaling laws interact with memory-intensive architectures to address the partial observability plateau in NetHack. Additionally, investigating the impact of expert data quality on the slope of the scaling law remains a key area for further research.
Summary updated