Masking Orchestration: Multi-task Pretraining for Multi-role Dialogue Representation Learning

Tianyi Wang, Yating Zhang, Xiaozhong Liu, Changlong Sun, Qiong Zhang

Abstract

Multi-role dialogue understanding comprises a wide range of diverse tasks such as question answering, act classification, and dialogue summarization. While dialogue corpora are abundantly available, labeled data for specific learning tasks can be highly scarce and expensive. In this work, we investigate dialogue context representation learning with various types of unsupervised pretraining tasks where the training objectives are given naturally according to the nature of the utterance and the structure of the multi-role conversation. Meanwhile, in order to locate essential information for dialogue summarization/extraction, the pretraining process enables external knowledge integration. The proposed fine-tuned pretraining mechanism is comprehensively evaluated via three different dialogue datasets along with a number of downstream dialogue-mining tasks. Results show that the proposed pretraining mechanism significantly contributes to all downstream tasks without discrimination to different encoders.

Introduction

Real-world conversations—such as courtroom debates, customer service calls, or corporate meetings—involve multiple roles with distinct goals, vocabularies, and power dynamics. Most existing NLP models are trained on monological text or simple two-party chats, failing to capture the complexities of multi-role interactions. This paper addresses three primary hurdles in multi-role dialogue mining:

  1. Role Divergence: Different participants (e.g., judge vs. defendant) use language differently. Standard models struggle to capture these persona-specific nuances.
  2. Contextual Dependency: Utterances in dialogue are often elliptical; a single word like "No" is meaningless without the preceding context.
  3. Data Scarcity: While raw transcripts are plentiful, labeled data for specialized tasks like legal fact recognition is expensive and rare.

To overcome these, the authors introduce a multi-task pretraining framework that "orchestrates" four unsupervised masking tasks to learn robust, context-aware dialogue representations.

Methods

The proposed architecture utilizes a hierarchical dialogue encoder. It employs a bidirectional LSTM (Bi-LSTM) for individual utterances and a combination of Bi-LSTM and a Transformer block to capture long-range dependencies across the entire dialogue.

Example of multi-task masking in a legal dialogue context.

The core innovation lies in the orchestration of four unsupervised pretraining tasks:

  • Word Prediction (W.P.): Predicts masked words using both local sentence context and global dialogue context.
  • Role Prediction (R.P.): Identifies the speaker's role based on utterance content, forcing the model to learn role-specific language patterns.
  • Sentence Generation (S.G.): An encoder-decoder task where the model reconstructs an entirely masked sentence to understand conversational flow.
  • Reference Prediction (F.P.): Specifically for the legal domain, the model predicts relevant external law articles, integrating domain-specific "world knowledge" into the representations.
Architecture of the multi-task pretraining framework.

Datasets and Evaluation

The framework was evaluated across three diverse domains:

  • Court Debate Dataset (CDD): 340,000 records of civil loan disputes involving four roles and external legal references.
  • Customer Service Dialogue (CSD): 5 million records from an e-commerce platform involving customers and agents.
  • English Meeting Dataset (EMD): A smaller collection of open datasets (AMI, Switchboard) used to test performance under data scarcity.

Key Results

The pretraining mechanism provided consistent gains across various encoders (HBLSTM, CRF-ASN):

  • Judicial Fact Recognition (JFR): On the CDD, the Macro F1 score improved from 39.61 to 45.45, demonstrating better performance on rare categories and superior noise filtering.
  • Controversy Focus Generation (CFG): For court case summarization, the Bleu-4 score nearly tripled, rising from 2.94 to 8.59.
  • Small Data Efficiency: On the EMD, pretraining led to a 7.6% to 56% increase in Bleu-4 scores, showing that the model converges faster and more accurately even with limited downstream data.

Conclusion

By treating speaker roles as anchors and integrating external knowledge through reference prediction, this multi-task masking strategy significantly enhances multi-role dialogue understanding. The results demonstrate that pretraining is particularly effective for filtering noise in complex dialogues and boosting performance in low-resource scenarios. Future work may involve incorporating more complex external resources, such as legal knowledge graphs, to further enrich representations.

Summary updated

Abstract

Multi-role dialogue understanding comprises a wide range of diverse tasks such as question answering, act classification, dialogue summarization etc. While dialogue corpora are abundantly available, labeled data, for specific learning tasks, can be highly scarce and expensive. In this work, we investigate dialogue context representation learning with various types unsupervised pretraining tasks where the training objectives are given naturally according to the nature of the utterance and the structure of the multi-role conversation. Meanwhile, in order to locate essential information for dialogue summarization/extraction, the pretraining process enables external knowledge integration. The proposed fine-tuned pretraining mechanism is comprehensively evaluated via three different dialogue datasets along with a number of downstream dialogue-mining tasks. Result shows that the proposed pretraining mechanism significantly contributes to all the downstream tasks without discrimination to different encoders.
BibTeX
@inproceedings{wang2020masking,
  author = {Tianyi Wang and Yating Zhang and Xiaozhong Liu and Changlong Sun and Qiong Zhang},
  title = {{Masking Orchestration: Multi-task Pretraining for Multi-role Dialogue Representation Learning}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2020},
  doi = {10.1609/aaai.v34i05.6459},
}
Figure