Ontologically Faithful Generation of Non-Player Character Dialogues

Nathaniel Weir, Ryan Thomas, Randolph D'Amore, Kellie Hill, Benjamin Van Durme, Harsh Jhamtani

Abstract

We introduce a language generation dataset grounded in a popular video game. KNUDGE (KNowledge Constrained User-NPC Dialogue GEneration) requires models to produce trees of dialogue between video game characters that accurately reflect quest and entity specifications stated in natural language. KNUDGE is constructed from side quest dialogues drawn directly from game data of Obsidian Entertainment’s The Outer Worlds, leading to real-world complexities in generation: (1) utterances must remain faithful to the game lore, including character personas and backstories; (2) a dialogue must accurately reveal new quest details to the human player; and (3) dialogues are large trees as opposed to linear chains of utterances. We report results for a set of neural generation models using supervised and in-context learning techniques; we find competent performance but room for future work addressing the challenges of creating realistic, game-quality dialogues.

The Challenge of Game Dialogue

In modern Role-Playing Games (RPGs), dialogue serves a functional purpose within the game's "ontology." An NPC’s lines must simultaneously reflect their personal backstory, adhere to the world’s history (lore), and provide the player with specific, actionable quest information.

Current Natural Language Generation (NLG) models often struggle with this "ontologically faithful" task. They may hallucinate facts, ignore quest objectives, or produce generic responses that break immersion. Furthermore, most existing research focuses on linear, one-on-one conversations, whereas real games utilize complex, branching dialogue trees where player choices lead to different narrative outcomes.

The KNUDGE task requires grounding dialogue in quest data and entity bios. Factual errors (in red) highlight the difficulty of maintaining ontological faithfulness.

The KNUDGE Dataset

The core contribution of this work is the KNUDGE (KNowledge constrained User-NPC Dialogue GEneration) dataset. Extracted directly from the game data of Obsidian Entertainment’s The Outer Worlds, it represents a significant step up in complexity from previous datasets.

  • Scale: 159 dialogue trees from 45 side quests, containing 4.7K utterances.
  • Richness: KNUDGE includes professionally written, branching trees paired with granular "support facts" pulled from fan wikis.
  • Complexity: On average, each tree contains 29 utterances and is constrained by over 1,300 tokens of lore and quest data.

Methodology: DialogueWriters

The authors propose DialogueWriters, a class of models designed for two primary scenarios: Next Utterance Prediction (NUP) and End-to-End (E2E) Generation. They tested these methods using both Supervised Learning (fine-tuning T5-large) and In-Context Learning (prompting GPT-3 and GPT-4).

To improve performance, a Knowledge Selection (KS) mechanism was introduced. Instead of feeding the model every possible fact, the KS module identifies specific facts from quest descriptions or character bios relevant to the current turn. This "factorized" approach helps the model focus on the correct constraints.

The DialogueWriter generation pipeline includes initial generation followed by revisions for lore usage and structure, with an automatic validator to ensure quality.

Key Insights and Results

  • The Functionality of Dialogue: Game dialogue is a "quest-serving" tool. If an NPC fails to mention a specific location or item required for the quest, the dialogue has failed its functional purpose.
  • The Copying Trade-off: Using Knowledge Selection significantly improved the models' ability to reflect lore by allowing them to "copy" specific text spans. However, this occasionally led to logical contradictions when facts were forced into ill-fitting contexts.
  • Human Evaluation: In Next Utterance Prediction, GPT-4 based models performed best (coherence score of 3.78/4.0) but still fell short of the human baseline (3.94).
  • Lore Usage: The KS variant of models scored significantly higher on lore usage (3.29/4.0) than models without it (2.76/4.0), proving that filtering relevant information is crucial for faithfulness.

Conclusion

While Large Language Models (LLMs) can generate fluent dialogue, they still struggle to match the coherence and narrative depth of professional human game writers. Future work will focus on improving persona embodiment and developing "copilot" tools to help human designers draft complex trees more efficiently.

Summary updated

Abstract

We introduce a language generation dataset grounded in a popular video game. KNUDGE (**KN**owledge Constrained **U**ser-NPC **D**ialogue **GE**neration) requires models to produce trees of dialogue between video game characters that accurately reflect quest and entity specifications stated in natural language. KNUDGE is constructed from side quest dialogues drawn directly from game data of Obsidian Entertainment’s _The Outer Worlds_, leading to real-world complexities in generation: (1) utterances must remain faithful to the game lore, including character personas and backstories; (2) a dialogue must accurately reveal new quest details to the human player; and (3) dialogues are large trees as opposed to linear chains of utterances. We report results for a set of neural generation models using supervised and in-context learning techniques; we find competent performance but room for future work addressing the challenges of creating realistic, game-quality dialogues.
BibTeX
@inproceedings{weir2024ontologically,
  author = {Nathaniel Weir and Ryan Thomas and Randolph D'Amore and Kellie Hill and Benjamin Van Durme and Harsh Jhamtani},
  title = {{Ontologically Faithful Generation of Non-Player Character Dialogues}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2024},
  url = {https://openreview.net/forum?id=VfUUool2UR},
}
Figure