TL;DR
This paper tackles the challenge of creating dynamic, coherent story experiences in open-world games by using reinforcement learning (RL) to train a "drama manager." This manager learns to subtly guide the game's narrative and character actions to achieve author-defined story goals, outperforming traditional search-based methods, especially in complex narratives.
What Problem Does This Paper Address?
Creating truly interactive and engaging story-driven games is a significant challenge. While many games offer rich, non-linear plots, players often feel like they're following a pre-written script or that their actions have little impact on the overarching narrative. The goal is to allow players significant agency while ensuring the story remains coherent and aligns with the author's artistic vision. Traditional approaches rely on local triggers and condition-action rules, which become unmanageable for complex stories. Declarative Optimization-Based Drama Management (DODM) offers a solution by allowing authors to specify what constitutes a good story (via an evaluation function) and leaving it to a drama manager (DM) to figure out how to achieve it. However, early search-based DM approaches struggled with scalability. This paper aims to improve upon these methods by applying reinforcement learning.
Methods
The core of this paper is the application of reinforcement learning to train a drama manager. The problem is framed as an RL task where the "state" is the current sequence of plot points and DM actions, and the "actions" are the DM's available commands to influence the game world (e.g., cause an event, deny an event, hint at something). The goal is to learn a policy that maximizes the author's evaluation function.
The authors compare three methods:
- SAS+ (Sampling Search): A search-based approach that explores a limited depth of possible future story states.
- TD Learning (Temporal-Difference Learning): A standard RL algorithm that learns state-value estimates to guide action selection. They use a neural network as a function approximator for these estimates.
- SASCE (Self-Adversarial / Self-Cooperative Exploration): This is the paper's novel contribution. It's a modified training paradigm for TD learning. Instead of training the RL agent on a player model that mimics the expected deployed player, SASCE trains on a player model that uses the drama manager's own current value estimates to select actions. This creates a feedback loop where the player can act cooperatively (leading to higher-rated states) or adversarially (leading to lower-rated states), forcing the drama manager to explore the state space more effectively and learn robust policies. They implement this with a mixed strategy, allowing for both cooperative and adversarial play during training.
Datasets
The authors evaluate their methods on two story domains:
- Tea for Three: A simplified version of an existing interactive fiction.
- Anchorhead: A more complex interactive fiction, used both as a subset and a larger portion.
These are existing story domains used in prior research, allowing for direct comparison of performance. No new benchmarks were introduced.
Key Insights and Ideas
- RL Outperforms Search in Complexity: Reinforcement learning, particularly with the SASCE training method, significantly outperforms traditional search-based methods in more complex narrative domains like Anchorhead. This suggests that RL's ability to learn long-term strategies is crucial for managing intricate stories.
- SASCE is Key for Robust Exploration: The SASCE training paradigm is critical. By training against a player model that leverages the DM's own value estimates, the system is forced to explore the state space more thoroughly and learn policies that are less susceptible to "dead ends" or suboptimal strategies.
- Mixed-Strategy Training is Optimal: For SASCE, a mixed strategy (combining cooperative and adversarial play during training) yielded the best results. Purely cooperative training was the least effective, suggesting that learning to handle potential "worst-case" scenarios is vital for robust performance.
- Authorial Pitfalls and DM Actions: The experiments on the larger Anchorhead portion highlight that even with advanced techniques, performance can be limited by the available DM actions. This suggests that authors need to carefully consider the set of tools they provide to the drama manager.
- The Value of Diversity: The paper touches on the potential issue of drama managers finding a narrow range of "good" stories and always steering towards them. This raises the question of whether evaluation functions should also incorporate a measure of story diversity to ensure replayability and emergent narrative.
Key Results
- Improved Performance: TD learning with SASCE consistently improved performance across both stories and evaluation metrics (mean and median story quality) compared to conventionally trained TD learning and SAS+ search.
- Anchorhead Subset: SASCE significantly improved the median performance on the Anchorhead subset, shifting the distribution of story qualities higher.
- Tea for Three: While SAS+ performed marginally better on the simpler Tea for Three story, TD learning (especially with SASCE) still showed strong results.
- Disagreement Analysis: Analysis of policy disagreements showed that search-based methods diverged from RL methods much earlier, indicating a difference in their strategic horizons.
Limitations and Future Work
- Player Model Abstraction: The current player models are abstractions of concrete actions. The authors acknowledge that a more realistic player model operating within the actual game world could lead to more accurate predictions and better policies.
- Authorial Affordances: The limited set of DM actions in some domains can hinder performance, suggesting that authors need to carefully design the available tools for the drama manager.
- Story Diversity: The authors suggest that future evaluation functions might need to incorporate story diversity as an explicit goal, rather than solely focusing on optimizing individual story quality.
- Real-World Player Testing: The ultimate validation would involve testing learned policies with actual human players to see how they perform in a real interactive environment.
- Robustness to Player Types: Investigating how policies trained under different player models perform against various types of human players is a key future direction.
Summary updated