TL;DR
This paper introduces a novel approach to creating highly realistic simulated players for interactive narrative systems using deep neural networks (RHNs and CNNs). These simulated players are crucial for training adaptive narrative planners, as they significantly reduce the need for large, real-world player datasets, leading to more generalizable narrative adaptation policies.
What Problem Does This Paper Address?
Interactive narrative systems aim to create engaging and personalized experiences by adapting the story in real-time based on player actions. Traditionally, training these adaptive systems relies on vast amounts of data from actual human players. However, collecting such data is resource-intensive and time-consuming. The core challenge this paper addresses is the scarcity of high-quality player data, which hinders the development of robust and generalizable interactive narrative planners. To overcome this, the authors propose creating "high-fidelity simulated players" that mimic human behavior, thereby enabling more efficient training and evaluation of narrative adaptation strategies.
Methods
The paper introduces two main deep neural network architectures for building these high-fidelity simulated players:
- Recurrent Highway Networks (RHNs) for Player Action Prediction: RHNs are a type of recurrent neural network that uses highway layers to enable greater depth and trainability. This architecture is designed to predict the player's next action within the narrative, capturing sequential dependencies in gameplay. The authors employ variational inference-based dropout for regularization, which helps improve generalization by applying a fixed dropout mask per sequence.
- Convolutional Neural Networks (CNNs) for Player Outcome Prediction: CNNs are used to predict the player's learning outcome (measured by Normalized Learning Gain - NLG). CNNs are well-suited for this task because they can identify repeatable patterns in gameplay features over time, without being strictly tied to exact input sequences like RNNs. The authors utilize a multi-task learning approach (predicting both outcome type and score) and a mixture-of-experts (ME) ensemble method to enhance the predictor's performance and robustness.
Datasets
The primary dataset used in this research was collected from two human subject studies involving the educational interactive narrative game, CRYSTAL ISLAND. This dataset comprises data from 402 players, including their gameplay action sequences, traits, interaction history, and pre/post-test outcomes. The dataset was collected using a narrative planner with a uniform random policy to broadly sample the planning policy space.
Key Insights and Ideas
- Deep NNs are Superior for Player Modeling: The research strongly suggests that deep neural networks (RHNs and CNNs) significantly outperform traditional shallow network models (like single-layer LSTMs) in accurately simulating player actions and predicting outcomes. This highlights the importance of network depth for learning complex player behaviors.
- RHNs Excel at Action Prediction: RHNs, with their ability to handle sequential data and increased depth, prove particularly effective for predicting player actions, outperforming even CNNs in this specific task. This is attributed to the rich information contained in the exact gameplay history sequence.
- CNNs are Robust for Outcome Prediction: CNNs demonstrate strong performance in predicting player outcomes, suggesting that while sequential history is important for actions, outcome prediction might benefit more from pattern recognition across time steps.
- Exploration is Key for Generalizable Narrative Policies: When using simulated players to train reinforcement learning (RL) agents for narrative planning, a more exploratory RL strategy leads to policies that are more adaptable and effective when interacting with unfamiliar player populations. Overly exploitative strategies result in brittle policies that perform poorly in novel environments.
Key Results
- Player Action Prediction: Deep RHN models achieved significantly higher accuracy and macro-average F1 scores compared to shallow LSTM and CNN models. For instance, the deep RHN model achieved an accuracy of 0.4435, outperforming the prior state-of-the-art shallow LSTM (0.3304).
- Player Outcome Prediction: CNN-based models, particularly those incorporating multi-task learning and mixture of experts, outperformed logistic regression and shallow LSTM models. The CNN model achieved an accuracy of 0.6096 and a macro-average F1 score of 0.6121.
- Narrative Planning Generalizability: In evaluations with increasingly unfamiliar simulated player models (Conditions 2 and 3), the A3C RL method, which is the most exploratory, consistently outperformed more exploitative methods like AQN and PPO. For example, in Condition 3, A3C achieved a policy value of 0.5510, while PPO achieved 0.4756, demonstrating A3C's superior ability to adapt to novel player behaviors.
Limitations and Future Work
- Limitations: The authors acknowledge that the evaluation of narrative planning policies was conducted using simulated players, not real humans. While high-fidelity simulation is the goal, it's not a perfect substitute for real-world interaction. The current work focuses on a specific educational game (CRYSTAL ISLAND), and its generalizability to other narrative types might need further investigation.
- Future Work: The authors propose extending this research by:
- Investigating the performance of simulated player modeling on a broader range of interactive narratives.
- Evaluating the effectiveness of data-driven interactive narrative planners, trained with high-fidelity simulated players, in real-time settings with human players.
Summary updated