Introduction
Document-grounded dialog systems (DGDS) aim to generate informative responses based on external knowledge sources. However, standard "direct" models often suffer from hallucinations, producing fluent but factually incorrect content. While existing solutions use discriminators or control tokens, these often require expensive human-labeled data or fail to model the logical connection between the document and the response. This paper proposes a Noisy Channel Model (NCM) that enforces factuality by design through a probabilistic reconstruction framework.
Methodology
The NCM decomposes the response generation probability $p(y | x, d)$ using Bayes' Theorem:
$$p(y | x, d) \propto p(d | y, x) \cdot p(y | x)$$
This factorization introduces two distinct components:
- Response Generation Model ($p(y | x)$): A standard language model trained on general dialog to ensure conversational fluency and coherence with the context ($x$).
- Channel Model ($p(d | y, x)$): A "verifier" that attempts to reconstruct the original grounding document ($d$) from the generated response ($y$) and context ($x$).
The intuition is that if a response is a faithful representation of the document, the document should be easier to reconstruct. Conversely, hallucinations make reconstruction difficult, resulting in a lower probability score.
Decoding and Control
To manage computational complexity, the authors employ two strategies: Reranking (scoring an N-best list) and Online Decoding (a modified beam search). A key advantage of this framework is controllability; by adjusting scaling factors ($\lambda$) for the channel model at inference time, users can tune the tradeoff between being "chatty" and being strictly factual without retraining the model.
Experimental Results
The model was evaluated across five datasets, including Wizard-of-Wikipedia (WoW), PersonaChat, and DSTC9. The primary metric for factuality was $Q^2$, a question-answering based metric that correlates well with human judgment.
Factuality and Beam Size
As illustrated in the figures below, the NCM consistently outperforms the direct model and reranking baselines in terms of factuality ($Q^2$).
In the Personachat dataset, the performance gap is even more pronounced. The NCM demonstrates a clear upward trend in factuality as the effective beam size increases, suggesting that a larger search space allows the channel model to identify more faithful hypotheses that the direct model might otherwise rank lower.
Conclusion
The Noisy Channel Model provides a robust, zero-shot framework for improving factuality in document-grounded dialog. By treating the document as a "source" to be reconstructed, the model inherently penalizes hallucinations. While online decoding increases computational overhead, the gains in factual consistency and the ability to tune the model at inference time represent a significant step forward for reliable conversational AI.
Summary updated