"It's Unwieldy and It Takes a Lot of Time." Challenges and Opportunities for Creating Agents in Commercial Games

Mikhail Jacob, Sam Devlin, Katja Hofmann

Introduction

While Reinforcement Learning (RL) has achieved landmark successes in games like Go and StarCraft, its adoption in commercial game development remains limited. This paper explores the "reality gap" between academic AI research and the practical needs of game developers. In the industry, the success of an agent is not defined by its ability to win, but by the quality of the player experience: Is it fun? Is its behavior readable? Can designers easily tune its personality?

Methodology

The researchers conducted a qualitative study involving semi-structured interviews with 17 industry professionals. These participants represented a broad spectrum of the industry, including AAA studios, indie developers, and industrial research labs, working on genres ranging from First-Person Shooters (FPS) to Real-Time Strategy (RTS) games.

Table 1: Participant metadata including roles and game genres.

The team employed thematic analysis to process over 21 hours of interview transcripts, iteratively coding the data to identify recurring challenges and compare them with existing academic literature.

The Workflow of Agent Creation

The study synthesizes these professional experiences into a taxonomy of challenges that span the entire development lifecycle, from initial design to final packaging.

Figure 1: A taxonomy of challenges across the game agent development lifecycle.

As shown in Figure 1, the process is non-linear and fraught with specific hurdles at each stage:

  • Design: Balancing designer control with emergent behavior. Challenges include maintaining "readability" and adhering to a specific game philosophy.
  • Implementation: Dealing with slow training times and the need for constant adaptation as the game evolves.
  • Evaluation: Moving beyond numeric reward functions to subjective "fun" and interpretability. "Reward hacking" remains a significant frustration for developers.

Key Challenges in Industry RL

The Control Paradox

Traditional game AI, like Behavior Trees, offers explicit control. RL is often viewed as a "black box," creating a barrier between a designer's intent and the resulting behavior. Designers need to be able to "nudge" agents without retraining them from scratch.

Optimal vs. Fun

In research, the goal is often to create an optimal agent. In games, an agent that is "too good" can be frustrating. The challenge lies in designing "sub-optimal" agents that provide an appropriate level of challenge and exhibit human-like flaws.

Technical Constraints

Academic RL often relies on accelerating game clocks. However, commercial game engines (physics, networking) often break at high speeds, leading to "unwieldy" training times that can stretch into weeks. Furthermore, training on "live" (and often buggy) development builds introduces significant instability rarely seen in stable academic benchmarks.

Future Opportunities

The paper identifies several research opportunities to bridge this gap. There is a critical need for better interpretability tools; if a designer cannot understand why an agent acted a certain way, they cannot trust it in production. Additionally, the authors advocate for hybrid systems that combine the authorial control of traditional methods with the power of machine learning.

Conclusion

Creating agents for commercial games is a complex, human-centric process. By shifting focus from pure performance to workflow-related issues like control, speed, and readability, the AI research community can develop tools that are truly transformative for the games industry.

Summary updated

Abstract

Game agents such as opponents, non-player characters, and teammates are central to player experiences in many modern games. As the landscape of AI techniques used in the games industry evolves to adopt machine learning (ML) more widely, it is vital that the research community learn from the best practices cultivated within the industry over decades creating agents. However, although commercial game agent creation pipelines are more mature than those based on ML, opportunities for improvement still abound. As a foundation for shared progress identifying research opportunities between researchers and practitioners, we interviewed seventeen game agent creators from AAA studios, indie studios, and industrial research labs about the challenges they experienced with their professional workflows. Our study revealed several open challenges ranging from design to implementation and evaluation. We compare with literature from the research community that address the challenges identified and conclude by highlighting promising directions for future research supporting agent creation in the games industry.
BibTeX
@inproceedings{jacob2020it,
  author = {Mikhail Jacob and Sam Devlin and Katja Hofmann},
  title = {{"It's Unwieldy and It Takes a Lot of Time." Challenges and Opportunities for Creating Agents in Commercial Games}},
  booktitle = {AIIDE},
  year = {2020},
  url = {https://openreview.net/forum?id=zop2ODL5ra},
}
Figure