Abstract
We introduce a language generation task grounded in a popular video game environment. KNUDGE (KNowledge Constrained User-NPC Dialogue GEneration) requires models to produce trees of dialogue between video game characters that accurately reflect quest and entity specifications stated in natural language. KNUDGE is constructed from side quest dialogues drawn directly from game data of Obsidian Entertainment's The Outer Worlds, leading to real-world complexities in generation: (1) dialogues are branching trees as opposed to linear chains of utterances; (2) utterances must remain faithful to the game lore -- character personas, backstories, and entity relationships; and (3) a dialogue must accurately reveal new quest details to the human player. We report results for a set of neural generation models using supervised and in-context learning techniques; we find competent performance but room for future work addressing the challenges of creating realistic, game-quality dialogues.
Introduction
In Role-Playing Games (RPGs), dialogue is the engine of the narrative. A single conversation with a non-player character (NPC) must simultaneously establish a character's persona, provide historical context (lore), and advance specific quest objectives, all while navigating a branching tree of player choices. Traditionally, this is a multi-million dollar manual authoring process.
This paper introduces KNUDGE (KNowledge Constrained User-NPC Dialogue GEneration), a framework and dataset designed to evaluate if large language models (LLMs) can handle the "ontological faithfulness" required for high-quality game writing.
The KNUDGE Dataset
The core contribution is the KNUDGE dataset, built from Obsidian Entertainment’s AAA RPG The Outer Worlds. Unlike existing dialogue research that focuses on linear "chit-chat," KNUDGE captures the non-linear, lore-heavy nature of professional game design.
The dataset consists of 159 dialogue trees from 45 side quests, featuring 4.7K utterances annotated with "support facts" pulled from fan-written wikis. On average, each dialogue is constrained by 1.3K tokens of quest and biographical data, representing a significantly higher complexity than previous benchmarks like LIGHT or TorchLight II.
Methodology: Achieving Ontological Faithfulness
The task is defined as mapping quest constraints ($Q$), biographical facts ($B$), and participant names ($P$) to a dialogue tree ($D$). The researchers explored two primary approaches:
- Supervised Learning (SL): Fine-tuning a T5-large model to generate the next utterance based on a linearized history of the conversation and the provided facts.
- In-Context Learning (ICL): Prompting GPT-3 (text-davinci-003) with zero-shot and few-shot examples.
A critical innovation is the Knowledge Selection (KS) mechanism. Because a single quest might involve hundreds of background facts, the KS model sub-selects only the most relevant information before generation, acting as a "mental filter" for the LLM.
Key Insights and Results
The study revealed several critical findings regarding the current state of AI in narrative generation:
- The "Copy-Paste" Trap: Supervised models (T5) often achieved high automatic scores by simply copying spans of text from the lore. However, human evaluators found this "bland" and "robotic."
- Fluency vs. Faithfulness: GPT-3 is incredibly fluent, but without explicit knowledge selection, it often hallucinates or ignores quest objectives. The KS-enhanced models were significantly better at using provided biography facts.
- The Importance of Filtering: Providing the model with too much lore actually hurts performance. Reducing the "noise" of irrelevant background facts allows the model to focus on specific details needed for the current turn.
- Human Preference: In head-to-head comparisons, humans preferred the ICL (GPT-3) outputs 81.2% of the time over other baselines for coherence and engagingness.
Conclusion
While LLMs show promise in generating fluent dialogue, hurdles remain in reasoning about the global structure of branching trees and capturing idiosyncratic character personas. Future work will likely focus on "copilot" tools that assist human designers by suggesting subtrees rather than replacing the creative role of the writer.
Summary updated