Controllable Factuality in Document-Grounded Dialog Systems Using a Noisy Channel Model

Nico Daheim, David Thulke, Christian Dugast, Hermann Ney

Abstract

In this work, we present a model for document-grounded response generation in dialog that is decomposed into two components according to Bayes theorem. One component is a traditional ungrounded response generation model and the other component models the reconstruction of the grounding document based on the dialog context and generated response. We propose different approximate decoding schemes and evaluate our approach on multiple open-domain and task-oriented document-grounded dialog datasets. Our experiments show that the model is more factual in terms of automatic factuality metrics than the baseline model. Furthermore, we outline how introducing scaling factors between the components allows for controlling the tradeoff between factuality and fluency in the model output. Finally, we compare our approach to a recently proposed method to control factuality in grounded dialog, CTRL (arXiv:2107.06963), and show that both approaches can be combined to achieve additional improvements.

Introduction

Document-grounded dialog systems aim to provide informative responses by leveraging external knowledge sources, such as Wikipedia articles or technical manuals. Despite their potential, these systems frequently suffer from "hallucinations"—generating responses that are fluent but factually inconsistent with the provided source document. Standard "direct" models typically attempt to learn response generation and document grounding simultaneously, often prioritizing conversational fluency at the expense of factual accuracy.

This research introduces a Noisy Channel Model approach to decouple these objectives. By factorizing the generation process, we can explicitly control the influence of the grounding document, allowing for a tunable balance between the "gift of gab" and adherence to the truth.

Methodology: The Noisy Channel Framework

The core of this approach lies in applying Bayes’ Theorem to the probability of a response. Traditionally, models calculate the direct probability $P(R | C, D)$, where $R$ is the response, $C$ is the context, and $D$ is the document. The Noisy Channel Model instead decomposes this into two distinct components:

  1. The Response Generation Model (The Prior): Represented as $P(R | C)$, this component functions as a traditional ungrounded chatbot. It ensures the response is coherent and flows naturally from the preceding conversation turns.
  2. The Channel Model (The Likelihood): Represented as $P(D | R, C)$, this component acts as a factual validator. It calculates the probability of the source document given the generated response and context. Effectively, it asks: "If an observer only saw this response, how likely is it that they could reconstruct the original source document?"

Controllable Decoding

To manage the interaction between these components, the authors introduce scaling factors ($\lambda_1, \lambda_2$). These parameters function as control knobs, allowing developers to increase the weight of the channel model to prioritize factuality or the prior model to prioritize fluency.

Because exact Bayesian inference is computationally prohibitive in large search spaces, two approximate decoding strategies are proposed:

  • Reranking: A standard model generates $N$ candidate responses, which are then re-scored and ranked using the Noisy Channel logic.
  • Online Decoding: The Noisy Channel score is integrated directly into the beam search process, influencing the selection of each token as the sentence is constructed.

Experimental Setup

The model's performance was evaluated across five diverse English datasets, covering both open-domain "chit-chat" and goal-oriented scenarios:

  • Wizard-of-Wikipedia (WoW): Conversations grounded in Wikipedia, including tests on "unseen" topics to measure generalization.
  • Doc2Dial & DSTC9: Task-oriented datasets focused on government services and FAQ-based assistance.
  • PersonaChat: Dialogs grounded in specific persona descriptions.
  • FaithDial: A refined version of WoW designed specifically to minimize hallucinations in training data.

Factuality was measured using $Q^2$ (a question-answering based metric) and BERTScore, while fluency was monitored via perplexity.

Key Insights and Results

The Factuality-Fluency Trade-off

The experiments confirmed a quantifiable trade-off: as the weight of the Channel Model increases, factuality metrics ($Q^2$) improve significantly. However, this often leads to higher perplexity, indicating that the model's language becomes less "natural" as it adheres more strictly to the source text. The Noisy Channel framework allows practitioners to select the optimal point on this curve for their specific application.

Performance and Generalization

The Noisy Channel model consistently outperformed direct baselines across all datasets. Notably, on the "unseen" portion of Wizard-of-Wikipedia, the model maintained high factuality, suggesting that the Bayesian decomposition helps the model generalize to new topics better than standard end-to-end training. Qualitative analysis showed the model was less prone to generic, safe responses (e.g., "I don't know") and more likely to incorporate specific, correct details from the grounding document.

Synergy with CTRL

The researchers also compared their approach to CTRL, a method using control tokens to guide model behavior. Interestingly, the Noisy Channel model and CTRL are complementary. When combined, they achieved higher factuality scores than either method produced in isolation, suggesting that structural decomposition and token-based guidance provide distinct benefits.

Conclusion

The Noisy Channel Model provides a robust framework for improving factuality in document-grounded dialog without requiring additional human-labeled data for discriminators. While the computational cost of online decoding remains a challenge for real-time production environments, the ability to explicitly tune the balance between fluency and factual grounding represents a significant step forward in building reliable conversational AI. Future work will focus on optimizing decoding efficiency and exploring more complex Bayesian factorizations.

Summary updated

Abstract

In this work, we present a model for document-grounded response generation in dialog that is decomposed into two components according to Bayes theorem. One component is a traditional ungrounded response generation model and the other component models the reconstruction of the grounding document based on the dialog context and generated response. We propose different approximate decoding schemes and evaluate our approach on multiple open-domain and task-oriented document-grounded dialog datasets. Our experiments show that the model is more factual in terms of automatic factuality metrics than the baseline model. Furthermore, we outline how introducing scaling factors between the components allows for controlling the tradeoff between factuality and fluency in the model output. Finally, we compare our approach to a recently proposed method to control factuality in grounded dialog, CTRL (arXiv:2107.06963), and show that both approaches can be combined to achieve additional improvements.
BibTeX
@inproceedings{daheim2022controllable,
  author = {Nico Daheim and David Thulke and Christian Dugast and Hermann Ney},
  title = {{Controllable Factuality in Document-Grounded Dialog Systems Using a Noisy Channel Model}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2022},
  url = {https://openreview.net/forum?id=4ex-rOS6Vo_},
}

Loading related content...

Figure