TL;DR
This paper introduces "threshold designer adaptation," a novel method for adapting AI co-creative systems to individual human designers. By retraining the AI model only after a certain threshold of user input, this approach leads to higher quality creative output and a more preferred user experience compared to a naive, continuous retraining strategy.
What Problem Does This Paper Address?
Creative AI systems are increasingly being developed to assist humans in artistic tasks, from music generation to game design. A critical challenge in these co-creative systems is "adaptability" – the ability of the AI to adjust its behavior to match the unique style and preferences of an individual human designer. Without adaptation, a single AI model might struggle to satisfy the diverse needs of different users, leading to suboptimal creative outcomes. While some co-creative systems exist, there's a lack of understanding regarding how and when to best adapt an AI model to a human designer, especially in domains requiring internal consistency and structure. This paper specifically addresses this gap within the context of game design, focusing on rhythm game chart creation.
Methods
The core contribution is threshold designer adaptation. This method determines a specific threshold ($\delta$) of collected user input data that must be met before the AI model is retrained. The optimal $\delta$ is found empirically by simulating co-creative interactions using existing human data. This simulation involves retraining the model on subsets of data and evaluating its performance on unseen data to find the $\delta$ that best approximates human interaction.
At runtime, the system starts with a pre-trained ML model. As a user interacts with the system, their input is collected as "retraining data." Once the amount of this data exceeds the calculated threshold $\delta$, the ML model is retrained to better match the user's style. This is a departure from a "naive" adaptation strategy, which retrains the model with every new piece of user data.
The authors implemented and evaluated this method using a co-creative rhythm game design tool called KiaiTime, designed for creating charts for the game Taiko no Tatsujin. KiaiTime uses a Long Short-Term Memory Recurrent Neural Network (LSTM RNN) called TaikoNation for generating note placements.
Datasets
The paper does not introduce new datasets for training or evaluation. Instead, it utilizes:
- Pre-trained TaikoNation model: This model was pre-trained on 100 high-quality human-authored charts from various designers. These charts were split into approximately 1.3 million training samples.
- Existing human data: Used during the grid search phase to empirically determine the optimal retraining threshold ($\delta$) and other hyperparameters.
- User-generated data: Collected during the human subject study from participants interacting with KiaiTime.
Key Insights and Ideas
- The "when" matters as much as the "how": Simply retraining an AI model continuously (naive approach) isn't always optimal. Threshold-based retraining allows the AI to gather enough context to make meaningful adaptations without being overly sensitive to minor fluctuations in user input.
- Empirical determination of adaptation thresholds is crucial: The optimal amount of data needed for retraining varies by domain and model architecture. The paper demonstrates a practical method for finding this threshold using existing data.
- Co-creative rhythm game design is a suitable domain for studying adaptation: The structured nature of rhythm games and the collaborative aspect of chart creation make it a good testbed for evaluating designer adaptation strategies.
- User preference aligns with better creative output: The study found that designers not only produced higher quality charts with threshold adaptation but also preferred interacting with the system using this method.
Key Results
The paper presents results from a human subject study with 20 participants.
- Quantitative Log Data (Table 1):
- Participants spent more time interacting with the threshold adaptation system (24.1 mins vs. 20.6 mins).
- Participants passed control to the AI more frequently with threshold adaptation (48.6 turns vs. 39.5 turns).
- Significantly more AI notes were kept by participants when using threshold adaptation (72.5% vs. 59.6%, p < 0.03).
- Charts created with threshold adaptation used significantly more patterns (21.72% vs. 18.83%, p < 0.05), indicating higher quality.
- Quantitative Survey Results (Table 2):
- Participants ranked threshold adaptation as significantly better in terms of "Enjoyed Most" (11 vs. 6).
- Threshold adaptation also received higher rankings for "Learned Better" (11 vs. 8) and "Higher Quality" (9 vs. 8), though these differences were not statistically significant.
- Qualitative Findings: Participants generally preferred the threshold adaptation strategy, finding it more helpful and less frustrating.
Limitations and Future Work
- Limitations:
- Single Domain Evaluation: The study was conducted solely in the rhythm game chart design domain. The authors acknowledge that evaluating the approach in other creative domains is necessary to assess its generalizability.
- "Correct" Threshold Value: While the chosen threshold value was determined empirically and showed good results, the authors note that further sweeping over a larger range of values and with more designers could refine this choice.
- Retraining Performance Impact: There's a potential concern that retraining might degrade ML model performance. While their results suggest otherwise in this specific case, it's an area to consider.
- Future Work:
- Cross-Domain Evaluation: Applying threshold designer adaptation to other creative domains (e.g., music composition, painting).
- Long-Term Adaptation: Investigating adaptation strategies that can be applied over extended periods (weeks or months) to better fit a designer's evolving tastes.
- Defining "Best Practice": Further research is needed to establish a more universal "best practice" for designer adaptation, as it is highly domain-specific.
- Addressing Ethical Concerns: Acknowledging the potential for AI to displace human creators and emphasizing that systems like KiaiTime are designed as tools to assist, not replace, human designers.
Summary updated