Masking Orchestration: Multi-Task Pretraining for Multi-Role Dialogue Representation Learning

Tianyi Wang, Yating Zhang, Xiaozhong Liu, Changlong Sun, Qiong Zhang

Introduction

Dialogue systems often struggle with multi-role scenarios, such as court trials or customer service interactions, where participants possess distinct goals, vocabularies, and social positions. Standard NLP models typically treat these dialogues as flat text sequences, failing to account for role dynamics and the influence of external knowledge (e.g., legal statutes). Furthermore, the scarcity of labeled data for these complex interactions hinders performance. This paper proposes a multi-task pretraining framework that "orchestrates" four unsupervised masking tasks to learn deep contextual representations from large-scale unlabeled dialogue data.

Methodology

The proposed framework utilizes a hierarchical encoder designed to capture both local utterance semantics and global dialogue context.

Hierarchical Encoder

  • Utterance Layer: Employs a Bi-LSTM to encode sentence-level semantics. Role embeddings are concatenated with word embeddings, allowing the model to differentiate word meanings based on the speaker's role.
  • Dialogue Layer: Uses a Bi-LSTM followed by a Transformer block to model long-term dependencies across the conversation.
  • Knowledge Enhance Layer: Integrates external references (e.g., law articles) via an attention mechanism to enrich dialogue representations.

Multi-Task Masking Strategy

To facilitate unsupervised learning, the authors introduce four masking tasks:

  1. Word Prediction (W.P.): Predicts randomly masked words within utterances, similar to BERT.
  2. Role Prediction (R.P.): Identifies the speaker's role for a given utterance.
  3. Sentence Generation (S.G.): Reconstructs a missing sentence using a Seq2Seq decoder based on surrounding context.
  4. Reference Prediction (F.P.): Predicts cited external knowledge, such as specific law articles, specifically for the legal domain.
Architecture of the multi-task pretraining framework, illustrating the hierarchical encoder and the four masking tasks.

Datasets

The model was evaluated on three distinct datasets to test its generalizability:

  • Court Debate Dataset (CDD): 340,000 civil trial records involving judges, plaintiffs, and defendants.
  • Customer Service Dialogue (CSD): 5 million e-commerce interaction records.
  • English Meeting Dataset (EMD): A composite of four public datasets (AMI, Switchboard, MRDA, bAbI) representing general-purpose corpora.

Results and Discussion

The pretrained models significantly outperformed vanilla (randomly initialized) versions and other state-of-the-art encoders across various downstream tasks.

Judicial Fact Recognition (JFR)

On the CDD dataset, the model achieved a Macro F1 score of 45.45, compared to 39.61 for the vanilla version. Ablation studies indicate that Word Prediction is the most critical task for this classification challenge, though the full orchestration of tasks yields the highest performance.

Controversy Focus Generation (CFG)

For court summarization, the BLEU-4 score improved by 192% (from 2.94 to 8.59) after pretraining. Sentence Generation was found to be vital for these generative tasks, highlighting the importance of cross-task synergy.

Ablation study results for JFR (Macro F1) and CFG (BLEU-4) tasks over training steps.

Key Insights

  • Role-Aware Embeddings: Integrating roles directly into word representations allows the model to capture nuanced intent (e.g., a judge's "No" vs. a defendant's "No").
  • Data Efficiency: Pretraining proved particularly beneficial for smaller datasets like EMD, accelerating convergence and improving final accuracy even when labeled data is scarce.

Conclusion

This work demonstrates that multi-task pretraining with role and knowledge awareness significantly enhances dialogue representation learning. While the Reference Prediction task is currently domain-specific, future work will explore adapting this approach to open-domain environments and incorporating more complex resources like knowledge graphs to further refine the Knowledge Enhance Layer.

Summary updated

Abstract

Multi-role dialogue understanding comprises a wide range of diverse tasks such as question answering, act classification, dialogue summarization etc. While dialogue corpora are abundantly available, labeled data, for specific learning tasks, can be highly scarce and expensive. In this work, we investigate dialogue context representation learning with various types unsupervised pretraining tasks where the training objectives are given naturally according to the nature of the utterance and the structure of the multi-role conversation. Meanwhile, in order to locate essential information for dialogue summarization/extraction, the pretraining process enables external knowledge integration. The proposed fine-tuned pretraining mechanism is comprehensively evaluated via three different dialogue datasets along with a number of downstream dialogue-mining tasks. Result shows that the proposed pretraining mechanism significantly contributes to all the downstream tasks without discrimination to different encoders.
BibTeX
@inproceedings{wang2020masking,
  author = {Tianyi Wang and Yating Zhang and Xiaozhong Liu and Changlong Sun and Qiong Zhang},
  title = {{Masking Orchestration: Multi-Task Pretraining for Multi-Role Dialogue Representation Learning}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2020},
  doi = {10.1609/aaai.v34i05.6459},
}
Figure