Controllable Factuality in Document-Grounded Dialog Systems Using a Noisy Channel Model

Nico Daheim, David Thulke, Christian Dugast, Hermann Ney

Introduction

Document-grounded dialog systems (DGDS) aim to generate informative responses based on external knowledge sources. However, standard "direct" models often suffer from hallucinations, producing fluent but factually incorrect content. While existing solutions use discriminators or control tokens, these often require expensive human-labeled data or fail to model the logical connection between the document and the response. This paper proposes a Noisy Channel Model (NCM) that enforces factuality by design through a probabilistic reconstruction framework.

Methodology

The NCM decomposes the response generation probability $p(y | x, d)$ using Bayes' Theorem:

$$p(y | x, d) \propto p(d | y, x) \cdot p(y | x)$$

This factorization introduces two distinct components:

  1. Response Generation Model ($p(y | x)$): A standard language model trained on general dialog to ensure conversational fluency and coherence with the context ($x$).
  2. Channel Model ($p(d | y, x)$): A "verifier" that attempts to reconstruct the original grounding document ($d$) from the generated response ($y$) and context ($x$).

The intuition is that if a response is a faithful representation of the document, the document should be easier to reconstruct. Conversely, hallucinations make reconstruction difficult, resulting in a lower probability score.

Decoding and Control

To manage computational complexity, the authors employ two strategies: Reranking (scoring an N-best list) and Online Decoding (a modified beam search). A key advantage of this framework is controllability; by adjusting scaling factors ($\lambda$) for the channel model at inference time, users can tune the tradeoff between being "chatty" and being strictly factual without retraining the model.

Experimental Results

The model was evaluated across five datasets, including Wizard-of-Wikipedia (WoW), PersonaChat, and DSTC9. The primary metric for factuality was $Q^2$, a question-answering based metric that correlates well with human judgment.

Factuality and Beam Size

As illustrated in the figures below, the NCM consistently outperforms the direct model and reranking baselines in terms of factuality ($Q^2$).

Factuality ($Q^2$) vs. beam size on the Wizard-of-Wikipedia (WoW) seen test set. The NCM (teal) significantly outperforms the direct model (orange).

In the Personachat dataset, the performance gap is even more pronounced. The NCM demonstrates a clear upward trend in factuality as the effective beam size increases, suggesting that a larger search space allows the channel model to identify more faithful hypotheses that the direct model might otherwise rank lower.

Factuality ($Q^2$) vs. beam size on the Personachat dataset, showing the NCM's superior performance and scaling with beam size.

Conclusion

The Noisy Channel Model provides a robust, zero-shot framework for improving factuality in document-grounded dialog. By treating the document as a "source" to be reconstructed, the model inherently penalizes hallucinations. While online decoding increases computational overhead, the gains in factual consistency and the ability to tune the model at inference time represent a significant step forward for reliable conversational AI.

Summary updated

Abstract

In this work, we present a model for document-grounded response generation in dialog that is decomposed into two components according to Bayes theorem. One component is a traditional ungrounded response generation model and the other component models the reconstruction of the grounding document based on the dialog context and generated response. We propose different approximate decoding schemes and evaluate our approach on multiple open-domain and task-oriented document-grounded dialog datasets. Our experiments show that the model is more factual in terms of automatic factuality metrics than the baseline model. Furthermore, we outline how introducing scaling factors between the components allows for controlling the tradeoff between factuality and fluency in the model output. Finally, we compare our approach to a recently proposed method to control factuality in grounded dialog, CTRL (arXiv:2107.06963), and show that both approaches can be combined to achieve additional improvements.
BibTeX
@inproceedings{daheim2022controllable,
  author = {Nico Daheim and David Thulke and Christian Dugast and Hermann Ney},
  title = {{Controllable Factuality in Document-Grounded Dialog Systems Using a Noisy Channel Model}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2022},
  url = {https://openreview.net/forum?id=4ex-rOS6Vo_},
}
Figure