Unstructured Text Enhanced Open-domain Dialogue System: A Systematic Survey

Longxuan Ma, Mingda Li, Weinan Zhang, Jiapeng Li, Ting LiuACM Trans. Information Systems 2022 · DOI · OpenReview

Introduction

Open-domain dialogue systems frequently encounter the "safe response" problem, where models produce generic, uninformative phrases like "I don't know" or "That's interesting." This occurs primarily because these models lack access to specific, grounded information during the generation process. While structured Knowledge Graphs (KGs) have been employed to mitigate this, they are often expensive to construct, difficult to scale, and too rigid for the fluid nature of human conversation.

Unstructured Text Enhanced Open-domain Dialogue Systems (UTEDS) offer a more scalable alternative by leveraging raw text—such as Wikipedia articles, news reports, and books—as an external knowledge source. However, grounding a conversation in raw text presents significant technical challenges, as models must navigate vast amounts of information to identify the single relevant fact that fits the current dialogue context. This survey organizes the research in this field into a coherent taxonomy of architectures and datasets.

Architectural Frameworks

The UTEDS landscape is primarily divided into two architectural paradigms: retrieval-based and generative models.

Retrieval-based Models

Retrieval-based models select the most appropriate response from a pre-defined set of candidates. These systems typically employ a three-stage pipeline: Fusion, Matching, and Ranking. In the Fusion stage, the dialogue history is combined with external text. The Matching stage compares this fused representation against potential response candidates, and the Ranking stage sorts these candidates to select the final output.

The general system component of Retrieval models in UTEDS, illustrating the pipeline from external knowledge and dialogue context to candidate ranking.

Generative Models

Generative models synthesize new responses word-by-word. Their architecture is more complex, focusing heavily on Knowledge Selection (KS). These models must encode the context, select the most relevant snippet of text (using either soft attention or hard selection), and then use a decoder—often equipped with a copy mechanism—to integrate that knowledge into the final response.

Dataset Taxonomy and Benchmarks

UTEDS research relies on specialized datasets categorized by the property of the text used, ranging from factual descriptions (e.g., Wikipedia) to fictional information (e.g., movie plots) and subjective comments (e.g., Reddit reviews).

Key benchmarks include:

  • Wizard of Wikipedia (WoW): Conversations where one participant has access to Wikipedia articles to inform their responses.
  • Persona-Chat: Dialogue grounded in short, descriptive "persona" sentences.
  • Holl-E & CMUDoG: Focused on movie-related discussions using reviews, plots, and scripts.

The following table provides a detailed comparison of these datasets, highlighting their knowledge sources, domains, and whether the knowledge is visible to one or both participants.

A comprehensive comparison of UTED datasets, detailing knowledge sources, dialogue sources, and domain coverage.

Core Challenges and Technical Insights

  • The Selection Bottleneck: The primary difficulty in UTEDS is not text generation, but knowledge selection. If the selection module identifies the wrong fact, the resulting response will be contextually inappropriate or factually incorrect.
  • Implicit vs. Explicit Selection: While early models used implicit selection (broad attention over a document), modern state-of-the-art models favor explicit selection, where the system first identifies a specific sentence or span before generating a response.
  • The Role of Copying: For generative models, a "copy mechanism"—which allows the model to point directly to words in the source text—is critical for maintaining factual accuracy and reducing hallucinations.
  • Pre-training Synergy: Large language models (LLMs) like GPT or BERT provide a strong foundation, but they require specialized adapter layers or memory networks to effectively utilize external, unstructured documents in a dialogue setting.

Future Research Directions

The survey identifies several "open frontiers" for future UTEDS research:

  • Knowledge Sparsity: Many current datasets utilize only a small fraction of the provided text, making it difficult for models to learn deep grounding.
  • Long-form Reasoning: Future systems must move beyond single-sentence retrieval to reason across entire documents for complex topics.
  • Evaluation Metrics: Standard metrics like BLEU and ROUGE are insufficient for measuring conversational quality. There is a critical need for "Knowledge-F1" metrics that specifically evaluate the accuracy of external text utilization.
  • Lifelong Learning: Intelligent agents should ideally retain and transfer knowledge across different sessions and domains over time, rather than treating each dialogue as an isolated event.

Summary updated

Abstract

Incorporating external knowledge into dialogue generation has been proven to benefit the performance of an open-domain Dialogue System (DS), such as generating informative or stylized responses, controlling conversation topics. In this article, we study the open-domain DS that uses unstructured text as external knowledge sources (\textbf{U}nstructured \textbf{T}ext \textbf{E}nhanced \textbf{D}ialogue \textbf{S}ystem, \textbf{UTEDS}). The existence of unstructured text entails distinctions between UTEDS and traditional data-driven DS and we aim to analyze these differences. We first give the definition of the UTEDS related concepts, then summarize the recently released datasets and models. We categorize UTEDS into Retrieval and Generative models and introduce them from the perspective of model components. The retrieval models consist of Fusion, Matching, and Ranking modules, while the generative models comprise Dialogue and Knowledge Encoding, Knowledge Selection, and Response Generation modules. We further summarize the evaluation methods utilized in UTEDS and analyze the current models' performance. At last, we discuss the future development trends of UTEDS, hoping to inspire new research in this field.
BibTeX
@article{ma2022unstructured,
  author = {Longxuan Ma and Mingda Li and Weinan Zhang and Jiapeng Li and Ting Liu},
  title = {{Unstructured Text Enhanced Open-domain Dialogue System: A Systematic Survey}},
  journal = {ACM Trans. Information Systems},
  year = {2022},
  doi = {10.1145/3464377},
}
Figure