Introduction
Deep Reinforcement Learning (DRL) has achieved remarkable milestones in gaming, from mastering Go to dominating complex real-time strategy environments. However, in the commercial gaming sector, the objective shifts from achieving "superhuman" dominance to fostering meaningful player engagement. In high-fidelity fighting games like Naruto Mobile, an unbeatable AI can lead to player frustration and churn. To address this, we present Shukai, a practical DRL agent system designed not just for competence, but for generalization and human-aligned interaction. Successfully deployed to a user base of over 100 million, Shukai demonstrates how DRL can serve as an effective training partner rather than a mere opponent.
The Challenge of Commercial Fighting Games
Implementing DRL in a live commercial environment like Naruto Mobile presents three primary hurdles:
- Scale: With over 400 distinct characters, training individual models for every possible matchup is computationally prohibitive.
- Generalization: Agents must maintain performance against hundreds of characters they have never encountered during training.
- Human Alignment: To be a useful sparring partner, an agent must exhibit human-like reaction times and diverse playstyles (e.g., aggressive vs. cautious) rather than frame-perfect, robotic execution.
The Shukai System Architecture
Shukai addresses these challenges through a combination of state quantification and a novel training framework.
1. Heterogeneous Agents
To prevent the AI from simply memorizing character-specific counters, the system utilizes three types of agents with varying levels of information access:
- Full-ID-State (FIS): A specialist agent that sees all character IDs and specific data.
- Quantitative-State (QS): An agent that sees its own ID but views the opponent only through numerical attributes (health, position, etc.).
- Full-Quantitative-State (FQS): A fully generalized agent that sees no IDs, relying entirely on raw numerical data.
By "imperfectizing" the input for QS and FQS agents, the system forces the AI to learn the fundamental "language" of the game—mechanics like spacing and timing—rather than character-specific exploits.
2. Heterogeneous League Training (HELT)
While standard League Training (as seen in AlphaStar) uses identical agents, Shukai introduces Heterogeneous League Training (HELT). In this setup, the FIS "specialist" acts as a League Exploiter. Because the FIS agent has full information, it quickly discovers optimal responses and "cheesy" strategies. This high-pressure environment forces the generalized agents (QS/FQS) to become more robust and adaptable to keep pace, significantly accelerating the learning process.
3. Aligning Behavior with Human Expectations
To ensure Shukai feels like a human opponent, the researchers implemented two key features:
- Action Constraints: The system uses action masks and reduced prediction frequencies to mimic human reaction times and physical limitations.
- Archetypal Rewards: Instead of a single objective, the team designed three distinct reward functions—Balanced, Cautious, and Aggressive—to mirror the playstyles found in human player data.
Experimental Results and Impact
Generalization and Efficiency
Shukai's ability to generalize is its most striking feature. Despite being trained on only 50 characters (roughly 13% of the roster), the QS agent maintained consistent competence across the entire 400+ character lineup. While the specialist FIS agent’s performance dropped by 36% when facing unfamiliar opponents, the QS agent remained stable. Furthermore, the HELT framework demonstrated a 22% improvement in sample efficiency compared to traditional homogeneous training methods.
Real-World Deployment
Since its integration into Naruto Mobile, Shukai has participated in over 500 million matches. The impact on player retention has been measurable, with a maximum increase of 5% in retention rates. Data suggests that Shukai serves as an effective teacher; beginner and intermediate players who sparred with the agent showed improved Elo scores in subsequent matches against other humans. In the live environment, Shukai maintained a next-day retention rate of over 50%, indicating high player acceptance.
Conclusion and Future Work
Shukai represents a shift in DRL research from pure optimization to practical, human-centric application. By prioritizing generalization through state quantification and leveraging the competitive pressure of HELT, the system provides a scalable solution for complex commercial games.
While Shukai excels at fundamental mechanics, it still faces challenges in mastering high-level, creative combos used by top-tier human players. Future research will focus on refining these complex tactical maneuvers and porting the HELT framework to other genres, such as first-person shooters, to further bridge the gap between artificial intelligence and human-like play.
Summary updated