Profile Consistency Identification for Open-domain Dialogue Agents

Haoyu Song, Yan Wang, Wei-Nan Zhang, Zhengyu Zhao, Ting Liu, Xiaojiang Liu

Introduction

Modern open-domain dialogue agents have achieved remarkable fluency, yet they frequently suffer from a lack of persona stability. A chatbot might claim to live in New York in one turn and then state it has never visited the East Coast in the next. This challenge, known as profile consistency, is critical for building trustworthy and engaging AI. While previous research has focused on generating responses based on a persona, less attention has been paid to the task of identifying whether a given response actually aligns with a structured profile.

Furthermore, industry-standard dialogue systems often store persona information in structured key-value pairs (e.g., Location: Beijing), whereas most existing consistency models are trained on natural language sentences. To bridge this gap, researchers have introduced KvPI, a large-scale dataset, and KvBERT, a hybrid model designed to reason over both linguistic and structural data.

The KvPI Dataset

The core contribution of this work is the Key-value Profile Identification (KvPI) dataset. Unlike previous datasets like PersonaChat, which use natural language premises, KvPI utilizes the structured format common in real-world applications.

Dataset Characteristics:

  • Scale: Contains over 118,540 human-annotated single-turn conversations.
  • Source: Derived from real-world interactions on Weibo, a major Chinese social media platform.
  • Structure: Each entry pairs a key-value profile (covering Gender, Location, and Constellation) with a query and a response.
  • Labeling: Responses are manually categorized into three classes: Entailed (consistent with the profile), Contradicted (conflicts with the profile), and Irrelevant (does not reveal profile-specific information).
Examples from the KvPI dataset demonstrating Entailed (E), Contradicted (C), and Irrelevant (I) labels across different domains.

The inclusion of the "Irrelevant" label is a key nuance. It allows the model to distinguish between a bot mentioning a location (e.g., "I like the history of Beijing") and a bot claiming to live there, which is essential for accurate consistency checking.

The KvBERT Model

To handle the dual nature of the task—understanding natural language and structured data—the authors proposed KvBERT (Key-value structure information enriched BERT). Standard language models like BERT often struggle with the inherent hierarchy of databases when the data is simply flattened into a string. KvBERT addresses this by using a hybrid architecture:

  1. Linguistic Processing: A 12-layer BERT-Base-Chinese model processes the linearized key-value pairs alongside the dialogue response to capture semantic meaning.
  2. Structural Reasoning: A 300D Tree-LSTM component explicitly models the dependency between values and their corresponding keys (e.g., mapping Beijing specifically to the Location attribute).
  3. Feature Fusion: The model fuses the linguistic features from BERT with the structural features from the Tree-LSTM before passing them to an output layer for classification.
The KvBERT architecture, combining a 12-layer BERT model for linguistic representation with a Tree-LSTM for structural key-value reasoning.

Key Insights and Results

Experimental results demonstrate that explicit structural modeling is vital. KvBERT achieved a state-of-the-art 91.7% accuracy on the KvPI test set, significantly outperforming standard BERT (88.0%) and TableBERT (88.6%).

Beyond raw accuracy, the model exhibits impressive "common sense" reasoning. For instance, if a profile lists Location: Beijing and the agent responds, "I'll show you around Tsinghua University," KvBERT correctly identifies this as Entailed. It understands the geographical relationship between the university and the city even without an explicit keyword match.

In practical applications, using KvBERT to rerank and filter responses from retrieval-based chatbots led to a substantial increase in persona-consistent replies and a sharp reduction in contradictions. Human evaluation confirmed these findings, with the model's predictions showing "almost perfect agreement" (Cohen’s Kappa > 0.8) with human annotators.

Conclusion and Future Work

KvBERT and the KvPI dataset represent a significant step toward more reliable dialogue agents. By treating profile consistency as a structured identification task, this research provides a robust framework for filtering inconsistent AI responses. Future work will look to expand the domain breadth beyond gender, location, and constellation to include more complex attributes like hobbies and professional backgrounds, as well as integrating this consistency-checking logic directly into the training phase of generative models.

Summary updated

BibTeX
@inproceedings{song2020profile,
  author = {Haoyu Song and Yan Wang and Wei-Nan Zhang and Zhengyu Zhao and Ting Liu and Xiaojiang Liu},
  title = {{Profile Consistency Identification for Open-domain Dialogue Agents}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2020},
  doi = {10.18653/v1/2020.emnlp-main.539},
}
Figure