The Turing Quest: Can Transformers Make Good NPCs?

Qi Chen Gao, Ali EmamiACL 2023

TL;DR

This paper introduces a novel pipeline for automatically generating believable non-playable character (NPC) dialogue for video games using Transformer-based language models like GPT-3. They also propose "The Turing Quest," a benchmark for evaluating NPC dialogue quality, demonstrating that their generated scripts can fool human judges into believing they were written by humans.

What Problem Does This Paper Address?

Creating immersive video game experiences increasingly relies on engaging interactions with non-playable characters (NPCs). However, manually writing dialogue for NPCs is a labor-intensive and time-consuming process, especially for large-scale games with many characters. This limits the depth and dynamism of NPC interactions, often confining them to scripted quest-givers. While large language models (LLMs) have shown promise in general conversational AI, their application to the specific, context-dependent domain of NPC dialogue generation, which requires adherence to game lore, character personality, and narrative, remains largely unexplored. The authors aim to bridge this gap by automating the creation of high-quality, believable NPC scripts.

Methods

The core of the paper is a three-module pipeline for NPC script generation:

  1. Feature Characterization Schema: This module defines a set of five key features to characterize an NPC: World Description, NPC Role, NPC Personality, Game State, and NPC Objective. These features capture crucial information about the NPC's context, purpose, and traits, allowing for a more nuanced representation than simple chatbots.
  2. Automatic Prompt Creation: Based on the characterization schema, this module generates tailored prompts for the LLM. These prompts provide the model with sufficient context about the NPC and the game to ensure generated dialogue is consistent and appropriate.
  3. Dialogue Generation: This module uses Transformer-based LLMs (specifically GPT-3 in their experiments) to generate dialogue iteratively, using the structured prompts and the ongoing dialogue history. They also implemented techniques to mitigate common LLM issues like repetition and to ensure coherent first sentences.

Additionally, they developed a "self-diagnosis" method where the LLM itself evaluates the generated dialogue against predefined metrics (coherence, believability, repetition, role alignment, and fittingness).

Datasets

The paper does not explicitly mention training datasets for the LLM itself, as they leverage pre-trained models like GPT-3. However, they introduce a new benchmark called The Turing Quest. This benchmark involves human judges evaluating generated NPC scripts to determine if they are distinguishable from human-written dialogue. They also used scripts from the game The Elder Scrolls V: Skyrim and a manually written script for comparison.

Key Insights and Ideas

  • NPCs are more than chatbots: The paper highlights that NPC dialogue generation is a distinct challenge from general chatbot generation. NPCs require strict adherence to game world lore, character personality, and narrative goals, which goes beyond simply maintaining a coherent conversation.
  • Structured characterization is key: The proposed feature characterization schema (World Desc., Role, Personality, Game State, Objective) is crucial for providing LLMs with the necessary context to generate believable and fitting NPC dialogue.
  • Prompt engineering matters: The effectiveness of the LLM is heavily influenced by the quality and detail of the prompts. Their "full prompt" approach, incorporating all characterization features, significantly outperformed a "minimal prompt" approach.
  • LLMs can fool humans: The Turing Quest results suggest that advanced LLMs, when properly prompted and guided, can generate NPC dialogue that is indistinguishable from human-written text for a significant portion of evaluators.
  • Self-diagnosis shows promise: The LLM's ability to self-evaluate dialogue quality, correlated with human judgment, offers a path towards automated evaluation and refinement of NPC scripts.

Key Results

  • The proposed pipeline, when applied to GPT-3, generated NPC scripts that were judged as human-written by 64.58% of participants on average in the Turing Quest.
  • The best performing generated script achieved a 75% pass rate in the Turing Quest.
  • Interestingly, 75% of judges incorrectly identified a manually written script as AI-generated, and 50% misidentified a script from Skyrim as AI-generated, suggesting evolving player expectations and the increasing capability of AI.
  • The "full prompt" approach resulted in an average score 4.06 points higher than the "minimal prompt" approach in their evaluation metric.
  • A Pearson correlation test showed a positive correlation between temperature settings and score, with higher temperatures (around 0.8-0.9) yielding better results.
  • The self-diagnosis module showed a strong positive correlation (r = .8092) with human-evaluated scores, indicating its reliability.

Limitations and Future Work

  • Player Dialogue Repetition: The dialogue generated for the player side of the conversation exhibited a higher degree of repetition and a tendency towards looping. The authors acknowledge this limitation, stating it was outside the scope of this paper, which focused solely on NPC dialogue.
  • Context Window Limitations: The maximum context window of the LLM can limit the dialogue history that can be processed, potentially causing the NPC to lose track of older information. A potential workaround is to discard older dialogue entries, though this might lead to information loss.
  • Bias in NPCs: The paper acknowledges the significant risk of bias in NPC models, especially given the large and young demographic of video game players. Addressing this bias is crucial for responsible AI development in games.
  • Future Work: The authors suggest exploring dialogue generation for dynamic NPCs with evolving roles or attributes, and further investigating the generation of player dialogue. They also mention the potential for modularity in their pipeline for generating responses.

Summary updated

BibTeX
@inproceedings{gao2023turing,
  author = {Qi Chen Gao and Ali Emami},
  title = {{The Turing Quest: Can Transformers Make Good NPCs?}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
}
Figure