DialogueBERT: A Self-Supervised Learning based Dialogue Pre-training Encoder

Zhenyu Zhang, Tao Guo, Meng Chen

Abstract

With the rapid development of artificial intelligence, conversational bots have became prevalent in mainstream E-commerce platforms, providing timely customer service. To satisfy users, these bots must understand intentions, detect emotions, and extract key entities. However, dialogue understanding is challenging due to alternating roles and hierarchical structures. We propose DialogueBERT, a contextual dialogue encoder based on BERT. We devise five self-supervised learning (SSL) tasks and integrate four input embeddings (turn, role, token, and position) to capture dialogue-specific relationships. Pre-trained on 70 million real-world dialogues and fine-tuned on three downstream tasks, DialogueBERT achieves 88.63% accuracy for intent recognition, 94.25% for emotion recognition, and a 97.04% F1 score for named entity recognition, significantly outperforming strong baselines.

Introduction

Standard language models like BERT are typically trained on "flat" text, such as Wikipedia or books, where the flow is linear and the voice is singular. In contrast, dialogues involve alternating roles (e.g., customer vs. agent), sequential turns, and a hierarchical structure where meaning depends heavily on preceding context. General-purpose models often struggle with these nuances. While generative models like DialoGPT exist for response creation, there is a gap in high-performance encoders specifically designed for dialogue understanding (NLU). DialogueBERT aims to bridge this gap by tailoring the Transformer architecture to the unique linguistic patterns of chat.

Methodology

DialogueBERT builds upon the standard BERT architecture but introduces several dialogue-centric innovations in its input representation and pooling strategy.

Multi-Modal Input Embeddings

To navigate the structure of a conversation, the model utilizes four distinct embeddings for every token:

  • Token & Position Embeddings: Standard BERT embeddings for lexical and sequence information.
  • Role Embedding: Identifies the speaker (e.g., "Customer" or "Agent").
  • Turn Embedding: Tracks the sequence of the conversation (Turn 1, Turn 2, etc.).

CNN Pooler and Architecture

Unlike standard BERT, which relies on a single [CLS] token for sequence-level representation, DialogueBERT employs a CNN Pooler. It applies six 1-D convolution kernels of varying sizes (5 to 20) followed by 2D pooling. This allows the model to capture local n-gram lexical and semantic information across the entire dialogue more effectively than a global token.

Figure 1: The architecture of DialogueBERT, highlighting the multi-modal embeddings, Transformer encoder, and the CNN pooler feeding into various SSL tasks.

Self-Supervised Pre-training Tasks

The model is pre-trained using five tasks designed to teach different aspects of conversation:

  1. Masked Language Modeling (MLM): Classic token-level "fill-in-the-blank."
  2. Masked Utterance Modeling (MUM): Reconstructing an entire masked utterance token-by-token.
  3. Replacement Discrimination (ReplDisc): Detecting if an utterance was replaced by a random one from another conversation.
  4. Turn Discrimination (TurnDisc): Detecting if two utterances within the same chat have been swapped.
  5. Response Selection (ResSel): Identifying the correct final response from a set of candidates.

Experiments and Datasets

The model was pre-trained on a massive, anonymized corpus of 70 million dialogues from JD.com’s e-commerce customer service platform. Evaluation was conducted on three downstream tasks:

  • Intent Recognition (IntR): Classifying 102 intent categories.
  • Emotion Recognition (EmoR): Detecting 5 emotion categories (Angry, Satisfied, Sad, Curious, Neutral).
  • Named Entity Recognition (NER): Extracting 28 types of entities.

Results and Discussion

DialogueBERT consistently outperformed strong baselines, including TOD-BERT and PT-CoDE, across all evaluated tasks.

Figure 3: Performance comparison of DialogueBERT against baseline models on Intent Recognition (IntR), Named Entity Recognition (NER), and Emotion Recognition (EmoR).

Key insights from the experimental results include:

  • MUM vs. MSM: Predicting the actual tokens of a masked utterance (MUM) proved more effective than selecting the correct utterance from a list (MSM), as it forces the model to learn finer-grained semantic structures.
  • CNN Pooler Efficacy: Ablation studies confirmed that the CNN pooler consistently outperformed standard [CLS] pooling, suggesting that dialogue features are better captured by local n-gram patterns.
  • Importance of MLM: Standard MLM remains essential for stable training, providing the baseline linguistic knowledge upon which dialogue-specific tasks build.

Conclusion

DialogueBERT demonstrates that specialized pre-training tasks and architectural adjustments, such as role/turn embeddings and CNN pooling, significantly enhance a model's ability to understand complex human conversations. Future work will explore optimized loss weighting for SSL tasks and the inclusion of query rewriting to further improve context handling.

Summary updated

Abstract

With the rapid development of artificial intelligence, conversational bots have became prevalent in mainstream E-commerce platforms, which can provide convenient customer service timely. To satisfy the user, the conversational bots need to understand the user's intention, detect the user's emotion, and extract the key entities from the conversational utterances. However, understanding dialogues is regarded as a very challenging task. Different from common language understanding, utterances in dialogues appear alternately from different roles and are usually organized as hierarchical structures. To facilitate the understanding of dialogues, in this paper, we propose a novel contextual dialogue encoder (i.e. DialogueBERT) based on the popular pre-trained language model BERT. Five self-supervised learning pre-training tasks are devised for learning the particularity of dialouge utterances. Four different input embeddings are integrated to catch the relationship between utterances, including turn embedding, role embedding, token embedding and position embedding. DialogueBERT was pre-trained with 70 million dialogues in real scenario, and then fine-tuned in three different downstream dialogue understanding tasks. Experimental results show that DialogueBERT achieves exciting results with 88.63% accuracy for intent recognition, 94.25% accuracy for emotion recognition and 97.04% F1 score for named entity recognition, which outperforms several strong baselines by a large margin.
BibTeX
@misc{zhang2021dialoguebert,
  author = {Zhenyu Zhang and Tao Guo and Meng Chen},
  title = {{DialogueBERT: A Self-Supervised Learning based Dialogue Pre-training Encoder}},
  year = {2021},
  eprint = {2109.10480},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2109.10480},
}
Figure