Advancing DRL Agents in Commercial Fighting Games: Training, Integration, and Agent-Human Alignment

Chen Zhang, Qiang He, Yuan Zhou, Elvis S. Liu, Hong Wang, Jian Zhao, Yang Wang

Introduction

Deep Reinforcement Learning (DRL) has achieved remarkable milestones in gaming, from mastering Go to dominating complex real-time strategy environments. However, in the commercial gaming sector, the objective shifts from achieving "superhuman" dominance to fostering meaningful player engagement. In high-fidelity fighting games like Naruto Mobile, an unbeatable AI can lead to player frustration and churn. To address this, we present Shukai, a practical DRL agent system designed not just for competence, but for generalization and human-aligned interaction. Successfully deployed to a user base of over 100 million, Shukai demonstrates how DRL can serve as an effective training partner rather than a mere opponent.

The Challenge of Commercial Fighting Games

Implementing DRL in a live commercial environment like Naruto Mobile presents three primary hurdles:

  1. Scale: With over 400 distinct characters, training individual models for every possible matchup is computationally prohibitive.
  2. Generalization: Agents must maintain performance against hundreds of characters they have never encountered during training.
  3. Human Alignment: To be a useful sparring partner, an agent must exhibit human-like reaction times and diverse playstyles (e.g., aggressive vs. cautious) rather than frame-perfect, robotic execution.

The Shukai System Architecture

Shukai addresses these challenges through a combination of state quantification and a novel training framework.

1. Heterogeneous Agents

To prevent the AI from simply memorizing character-specific counters, the system utilizes three types of agents with varying levels of information access:

  • Full-ID-State (FIS): A specialist agent that sees all character IDs and specific data.
  • Quantitative-State (QS): An agent that sees its own ID but views the opponent only through numerical attributes (health, position, etc.).
  • Full-Quantitative-State (FQS): A fully generalized agent that sees no IDs, relying entirely on raw numerical data.

By "imperfectizing" the input for QS and FQS agents, the system forces the AI to learn the fundamental "language" of the game—mechanics like spacing and timing—rather than character-specific exploits.

2. Heterogeneous League Training (HELT)

While standard League Training (as seen in AlphaStar) uses identical agents, Shukai introduces Heterogeneous League Training (HELT). In this setup, the FIS "specialist" acts as a League Exploiter. Because the FIS agent has full information, it quickly discovers optimal responses and "cheesy" strategies. This high-pressure environment forces the generalized agents (QS/FQS) to become more robust and adaptable to keep pace, significantly accelerating the learning process.

3. Aligning Behavior with Human Expectations

To ensure Shukai feels like a human opponent, the researchers implemented two key features:

  • Action Constraints: The system uses action masks and reduced prediction frequencies to mimic human reaction times and physical limitations.
  • Archetypal Rewards: Instead of a single objective, the team designed three distinct reward functions—Balanced, Cautious, and Aggressive—to mirror the playstyles found in human player data.

Experimental Results and Impact

Generalization and Efficiency

Shukai's ability to generalize is its most striking feature. Despite being trained on only 50 characters (roughly 13% of the roster), the QS agent maintained consistent competence across the entire 400+ character lineup. While the specialist FIS agent’s performance dropped by 36% when facing unfamiliar opponents, the QS agent remained stable. Furthermore, the HELT framework demonstrated a 22% improvement in sample efficiency compared to traditional homogeneous training methods.

Real-World Deployment

Since its integration into Naruto Mobile, Shukai has participated in over 500 million matches. The impact on player retention has been measurable, with a maximum increase of 5% in retention rates. Data suggests that Shukai serves as an effective teacher; beginner and intermediate players who sparred with the agent showed improved Elo scores in subsequent matches against other humans. In the live environment, Shukai maintained a next-day retention rate of over 50%, indicating high player acceptance.

Conclusion and Future Work

Shukai represents a shift in DRL research from pure optimization to practical, human-centric application. By prioritizing generalization through state quantification and leveraging the competitive pressure of HELT, the system provides a scalable solution for complex commercial games.

While Shukai excels at fundamental mechanics, it still faces challenges in mastering high-level, creative combos used by top-tier human players. Future research will focus on refining these complex tactical maneuvers and porting the HELT framework to other genres, such as first-person shooters, to further bridge the gap between artificial intelligence and human-like play.

Summary updated

Abstract

Deep Reinforcement Learning (DRL) agents have demonstrated impressive success in a wide range of game genres. However, existing research primarily focuses on optimizing DRL competence rather than addressing the challenge of prolonged player interaction. In this paper, we propose a practical DRL agent system for fighting games named Sh\=ukai, which has been successfully deployed to Naruto Mobile, a popular fighting game with over 100 million registered users. Sh\=ukai quantifies the state to enhance generalizability, introducing Heterogeneous League Training (HELT) to achieve balanced competence, generalizability, and training efficiency. Furthermore, Sh\=ukai implements specific rewards to align the agent's behavior with human expectations. Sh\=ukai's ability to generalize is demonstrated by its consistent competence across all characters, even though it was trained on only 13% of them. Additionally, HELT exhibits a remarkable 22% improvement in sample efficiency. Sh\=ukai serves as a valuable training partner for players in Naruto Mobile, enabling them to enhance their abilities and skills.
BibTeX
@inproceedings{zhang2024advancing,
  author = {Chen Zhang and Qiang He and Yuan Zhou and Elvis S. Liu and Hong Wang and Jian Zhao and Yang Wang},
  title = {{Advancing DRL Agents in Commercial Fighting Games: Training, Integration, and Agent-Human Alignment}},
  booktitle = {International Conference on Machine Learning},
  year = {2024},
  url = {https://openreview.net/forum?id=eN1T7I7OpZ},
}
Figure