Abstract
We introduce a language generation dataset grounded in a popular video game. KNUDGE (KNowledge Constrained User-NPC Dialogue GEneration) requires models to produce trees of dialogue between video game characters that accurately reflect quest and entity specifications stated in natural language. KNUDGE is constructed from side quest dialogues drawn directly from game data of Obsidian Entertainment’s The Outer Worlds, leading to real-world complexities in generation: (1) utterances must remain faithful to the game lore, including character personas and backstories; (2) a dialogue must accurately reveal new quest details to the human player; and (3) dialogues are large trees as opposed to linear chains of utterances. We report results for a set of neural generation models using supervised and in-context learning techniques; we find competent performance but room for future work addressing the challenges of creating realistic, game-quality dialogues.
The Challenge of Game Dialogue
In modern Role-Playing Games (RPGs), dialogue serves a functional purpose within the game's "ontology." An NPC’s lines must simultaneously reflect their personal backstory, adhere to the world’s history (lore), and provide the player with specific, actionable quest information.
Current Natural Language Generation (NLG) models often struggle with this "ontologically faithful" task. They may hallucinate facts, ignore quest objectives, or produce generic responses that break immersion. Furthermore, most existing research focuses on linear, one-on-one conversations, whereas real games utilize complex, branching dialogue trees where player choices lead to different narrative outcomes.
The KNUDGE Dataset
The core contribution of this work is the KNUDGE (KNowledge constrained User-NPC Dialogue GEneration) dataset. Extracted directly from the game data of Obsidian Entertainment’s The Outer Worlds, it represents a significant step up in complexity from previous datasets.
- Scale: 159 dialogue trees from 45 side quests, containing 4.7K utterances.
- Richness: KNUDGE includes professionally written, branching trees paired with granular "support facts" pulled from fan wikis.
- Complexity: On average, each tree contains 29 utterances and is constrained by over 1,300 tokens of lore and quest data.
Methodology: DialogueWriters
The authors propose DialogueWriters, a class of models designed for two primary scenarios: Next Utterance Prediction (NUP) and End-to-End (E2E) Generation. They tested these methods using both Supervised Learning (fine-tuning T5-large) and In-Context Learning (prompting GPT-3 and GPT-4).
To improve performance, a Knowledge Selection (KS) mechanism was introduced. Instead of feeding the model every possible fact, the KS module identifies specific facts from quest descriptions or character bios relevant to the current turn. This "factorized" approach helps the model focus on the correct constraints.
Key Insights and Results
- The Functionality of Dialogue: Game dialogue is a "quest-serving" tool. If an NPC fails to mention a specific location or item required for the quest, the dialogue has failed its functional purpose.
- The Copying Trade-off: Using Knowledge Selection significantly improved the models' ability to reflect lore by allowing them to "copy" specific text spans. However, this occasionally led to logical contradictions when facts were forced into ill-fitting contexts.
- Human Evaluation: In Next Utterance Prediction, GPT-4 based models performed best (coherence score of 3.78/4.0) but still fell short of the human baseline (3.94).
- Lore Usage: The KS variant of models scored significantly higher on lore usage (3.29/4.0) than models without it (2.76/4.0), proving that filtering relevant information is crucial for faithfulness.
Conclusion
While Large Language Models (LLMs) can generate fluent dialogue, they still struggle to match the coherence and narrative depth of professional human game writers. Future work will focus on improving persona embodiment and developing "copilot" tools to help human designers draft complex trees more efficiently.
Summary updated