Abstract
With the rapid development of artificial intelligence, conversational bots have became prevalent in mainstream E-commerce platforms, providing timely customer service. To satisfy users, these bots must understand intentions, detect emotions, and extract key entities. However, dialogue understanding is challenging due to alternating roles and hierarchical structures. We propose DialogueBERT, a contextual dialogue encoder based on BERT. We devise five self-supervised learning (SSL) tasks and integrate four input embeddings (turn, role, token, and position) to capture dialogue-specific relationships. Pre-trained on 70 million real-world dialogues and fine-tuned on three downstream tasks, DialogueBERT achieves 88.63% accuracy for intent recognition, 94.25% for emotion recognition, and a 97.04% F1 score for named entity recognition, significantly outperforming strong baselines.
Introduction
Standard language models like BERT are typically trained on "flat" text, such as Wikipedia or books, where the flow is linear and the voice is singular. In contrast, dialogues involve alternating roles (e.g., customer vs. agent), sequential turns, and a hierarchical structure where meaning depends heavily on preceding context. General-purpose models often struggle with these nuances. While generative models like DialoGPT exist for response creation, there is a gap in high-performance encoders specifically designed for dialogue understanding (NLU). DialogueBERT aims to bridge this gap by tailoring the Transformer architecture to the unique linguistic patterns of chat.
Methodology
DialogueBERT builds upon the standard BERT architecture but introduces several dialogue-centric innovations in its input representation and pooling strategy.
Multi-Modal Input Embeddings
To navigate the structure of a conversation, the model utilizes four distinct embeddings for every token:
- Token & Position Embeddings: Standard BERT embeddings for lexical and sequence information.
- Role Embedding: Identifies the speaker (e.g., "Customer" or "Agent").
- Turn Embedding: Tracks the sequence of the conversation (Turn 1, Turn 2, etc.).
CNN Pooler and Architecture
Unlike standard BERT, which relies on a single [CLS] token for sequence-level representation, DialogueBERT employs a CNN Pooler. It applies six 1-D convolution kernels of varying sizes (5 to 20) followed by 2D pooling. This allows the model to capture local n-gram lexical and semantic information across the entire dialogue more effectively than a global token.
Self-Supervised Pre-training Tasks
The model is pre-trained using five tasks designed to teach different aspects of conversation:
- Masked Language Modeling (MLM): Classic token-level "fill-in-the-blank."
- Masked Utterance Modeling (MUM): Reconstructing an entire masked utterance token-by-token.
- Replacement Discrimination (ReplDisc): Detecting if an utterance was replaced by a random one from another conversation.
- Turn Discrimination (TurnDisc): Detecting if two utterances within the same chat have been swapped.
- Response Selection (ResSel): Identifying the correct final response from a set of candidates.
Experiments and Datasets
The model was pre-trained on a massive, anonymized corpus of 70 million dialogues from JD.com’s e-commerce customer service platform. Evaluation was conducted on three downstream tasks:
- Intent Recognition (IntR): Classifying 102 intent categories.
- Emotion Recognition (EmoR): Detecting 5 emotion categories (Angry, Satisfied, Sad, Curious, Neutral).
- Named Entity Recognition (NER): Extracting 28 types of entities.
Results and Discussion
DialogueBERT consistently outperformed strong baselines, including TOD-BERT and PT-CoDE, across all evaluated tasks.
Key insights from the experimental results include:
- MUM vs. MSM: Predicting the actual tokens of a masked utterance (MUM) proved more effective than selecting the correct utterance from a list (MSM), as it forces the model to learn finer-grained semantic structures.
- CNN Pooler Efficacy: Ablation studies confirmed that the CNN pooler consistently outperformed standard
[CLS]pooling, suggesting that dialogue features are better captured by local n-gram patterns. - Importance of MLM: Standard MLM remains essential for stable training, providing the baseline linguistic knowledge upon which dialogue-specific tasks build.
Conclusion
DialogueBERT demonstrates that specialized pre-training tasks and architectural adjustments, such as role/turn embeddings and CNN pooling, significantly enhance a model's ability to understand complex human conversations. Future work will explore optimized loss weighting for SSL tasks and the inclusion of query rewriting to further improve context handling.
Summary updated