Profile Consistency Identification for Open-domain Dialogue Agents
Introduction
Maintaining a consistent persona is a significant challenge for open-domain dialogue agents. While modern AI can generate fluent text, agents often suffer from "personality splits," where a response contradicts previously established attributes. This research addresses the problem of identifying consistency between dialogue responses and structured persona profiles.
The Challenge of Structured Profiles
Real-world agent profiles are typically stored as structured key-value pairs (e.g., Location: Beijing, Gender: Female). Standard Natural Language Inference (NLI) models, designed for sentence-to-sentence logic, struggle to reason over this structured data. They often fail to distinguish between responses that are logically entailed by a profile, those that contradict it, and those that are merely irrelevant.
The KvPI Dataset
To address this, the authors introduced the Key-value Profile Identification (KvPI) dataset. It consists of over 118,000 single-turn conversations. Each entry includes a key-value profile (covering Gender, Location, and Constellation), a query, and a response.
Human annotators categorized responses into three classes:
- Entailed: The response is consistent with the profile.
- Contradicted: The response conflicts with the profile.
- Irrelevant: The response mentions profile-related topics but does not reveal the agent's own attributes (e.g., "I love Beijing" vs "I live in Beijing").
The KvBERT Model
The proposed KvBERT model is a hybrid architecture designed to bridge the gap between unstructured text and structured data by fusing sequential and hierarchical representations.
The architecture consists of two main branches:
- BERT Component: A 12-layer BERT-base-Chinese model processes a linearized version of the key-value pairs alongside the dialogue response to capture rich contextual information.
- Tree-LSTM Component: To explicitly capture the hierarchical dependency of the profile (e.g., linking the value
Beijingto the keyLocation), a 300D Tree-LSTM is employed. The dialogue response is also parsed into a dependency tree and encoded via a separate Tree-LSTM.
The representations from both branches are fused using element-wise operations (multiplication, difference, and concatenation) before a final classification layer determines the consistency label.
Key Insights and Results
- Superior Performance: KvBERT achieved 91.7% accuracy on the KvPI test set, significantly outperforming standard BERT (88.0%) and TableBERT (88.6%).
- Structural Importance: Explicitly modeling the key-value hierarchy with a Tree-LSTM is crucial. However, the authors noted an "80% threshold": the Tree-LSTM only improves the overall model once its own structural encoding accuracy reaches approximately 80%. Below this level, it can introduce noise that degrades performance.
- Real-world Utility: When used to rerank responses for a retrieval-based chatbot, KvBERT increased the rate of consistent responses by 10-20%. It also showed high agreement with human labels (Cohen’s Kappa > 0.8), suggesting it can serve as a reliable automated metric for persona consistency.
Conclusion
KvBERT demonstrates that incorporating structural priors into transformer-based models is essential for reasoning over structured persona data. While currently limited to specific domains like location and gender, this approach paves the way for more reliable and consistent long-term dialogue agents.
Summary updated