Profile Consistency Identification for Open-domain Dialogue Agents

Haoyu Song, Yan Wang, Wei-Nan Zhang, Zhengyu Zhao, Ting Liu, Xiaojiang Liu

Profile Consistency Identification for Open-domain Dialogue Agents

Introduction

Maintaining a consistent persona is a significant challenge for open-domain dialogue agents. While modern AI can generate fluent text, agents often suffer from "personality splits," where a response contradicts previously established attributes. This research addresses the problem of identifying consistency between dialogue responses and structured persona profiles.

The Challenge of Structured Profiles

Real-world agent profiles are typically stored as structured key-value pairs (e.g., Location: Beijing, Gender: Female). Standard Natural Language Inference (NLI) models, designed for sentence-to-sentence logic, struggle to reason over this structured data. They often fail to distinguish between responses that are logically entailed by a profile, those that contradict it, and those that are merely irrelevant.

The KvPI Dataset

To address this, the authors introduced the Key-value Profile Identification (KvPI) dataset. It consists of over 118,000 single-turn conversations. Each entry includes a key-value profile (covering Gender, Location, and Constellation), a query, and a response.

Example of profile-based dialogue consistency identification with entailed and contradicted responses.

Human annotators categorized responses into three classes:

  • Entailed: The response is consistent with the profile.
  • Contradicted: The response conflicts with the profile.
  • Irrelevant: The response mentions profile-related topics but does not reveal the agent's own attributes (e.g., "I love Beijing" vs "I live in Beijing").

The KvBERT Model

The proposed KvBERT model is a hybrid architecture designed to bridge the gap between unstructured text and structured data by fusing sequential and hierarchical representations.

The KvBERT architecture, fusing a 12-layer BERT model with a 300D Tree-LSTM to process structured profiles and dialogue responses.

The architecture consists of two main branches:

  1. BERT Component: A 12-layer BERT-base-Chinese model processes a linearized version of the key-value pairs alongside the dialogue response to capture rich contextual information.
  2. Tree-LSTM Component: To explicitly capture the hierarchical dependency of the profile (e.g., linking the value Beijing to the key Location), a 300D Tree-LSTM is employed. The dialogue response is also parsed into a dependency tree and encoded via a separate Tree-LSTM.

The representations from both branches are fused using element-wise operations (multiplication, difference, and concatenation) before a final classification layer determines the consistency label.

Key Insights and Results

  • Superior Performance: KvBERT achieved 91.7% accuracy on the KvPI test set, significantly outperforming standard BERT (88.0%) and TableBERT (88.6%).
  • Structural Importance: Explicitly modeling the key-value hierarchy with a Tree-LSTM is crucial. However, the authors noted an "80% threshold": the Tree-LSTM only improves the overall model once its own structural encoding accuracy reaches approximately 80%. Below this level, it can introduce noise that degrades performance.
  • Real-world Utility: When used to rerank responses for a retrieval-based chatbot, KvBERT increased the rate of consistent responses by 10-20%. It also showed high agreement with human labels (Cohen’s Kappa > 0.8), suggesting it can serve as a reliable automated metric for persona consistency.

Conclusion

KvBERT demonstrates that incorporating structural priors into transformer-based models is essential for reasoning over structured persona data. While currently limited to specific domains like location and gender, this approach paves the way for more reliable and consistent long-term dialogue agents.

Summary updated

BibTeX
@inproceedings{song2020profile,
  author = {Haoyu Song and Yan Wang and Wei-Nan Zhang and Zhengyu Zhao and Ting Liu and Xiaojiang Liu},
  title = {{Profile Consistency Identification for Open-domain Dialogue Agents}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2020},
  doi = {10.18653/v1/2020.emnlp-main.539},
}
Figure