Introduction
Dialogue systems often struggle with multi-role scenarios, such as court trials or customer service interactions, where participants possess distinct goals, vocabularies, and social positions. Standard NLP models typically treat these dialogues as flat text sequences, failing to account for role dynamics and the influence of external knowledge (e.g., legal statutes). Furthermore, the scarcity of labeled data for these complex interactions hinders performance. This paper proposes a multi-task pretraining framework that "orchestrates" four unsupervised masking tasks to learn deep contextual representations from large-scale unlabeled dialogue data.
Methodology
The proposed framework utilizes a hierarchical encoder designed to capture both local utterance semantics and global dialogue context.
Hierarchical Encoder
- Utterance Layer: Employs a Bi-LSTM to encode sentence-level semantics. Role embeddings are concatenated with word embeddings, allowing the model to differentiate word meanings based on the speaker's role.
- Dialogue Layer: Uses a Bi-LSTM followed by a Transformer block to model long-term dependencies across the conversation.
- Knowledge Enhance Layer: Integrates external references (e.g., law articles) via an attention mechanism to enrich dialogue representations.
Multi-Task Masking Strategy
To facilitate unsupervised learning, the authors introduce four masking tasks:
- Word Prediction (W.P.): Predicts randomly masked words within utterances, similar to BERT.
- Role Prediction (R.P.): Identifies the speaker's role for a given utterance.
- Sentence Generation (S.G.): Reconstructs a missing sentence using a Seq2Seq decoder based on surrounding context.
- Reference Prediction (F.P.): Predicts cited external knowledge, such as specific law articles, specifically for the legal domain.
Datasets
The model was evaluated on three distinct datasets to test its generalizability:
- Court Debate Dataset (CDD): 340,000 civil trial records involving judges, plaintiffs, and defendants.
- Customer Service Dialogue (CSD): 5 million e-commerce interaction records.
- English Meeting Dataset (EMD): A composite of four public datasets (AMI, Switchboard, MRDA, bAbI) representing general-purpose corpora.
Results and Discussion
The pretrained models significantly outperformed vanilla (randomly initialized) versions and other state-of-the-art encoders across various downstream tasks.
Judicial Fact Recognition (JFR)
On the CDD dataset, the model achieved a Macro F1 score of 45.45, compared to 39.61 for the vanilla version. Ablation studies indicate that Word Prediction is the most critical task for this classification challenge, though the full orchestration of tasks yields the highest performance.
Controversy Focus Generation (CFG)
For court summarization, the BLEU-4 score improved by 192% (from 2.94 to 8.59) after pretraining. Sentence Generation was found to be vital for these generative tasks, highlighting the importance of cross-task synergy.
Key Insights
- Role-Aware Embeddings: Integrating roles directly into word representations allows the model to capture nuanced intent (e.g., a judge's "No" vs. a defendant's "No").
- Data Efficiency: Pretraining proved particularly beneficial for smaller datasets like EMD, accelerating convergence and improving final accuracy even when labeled data is scarce.
Conclusion
This work demonstrates that multi-task pretraining with role and knowledge awareness significantly enhances dialogue representation learning. While the Reference Prediction task is currently domain-specific, future work will explore adapting this approach to open-domain environments and incorporating more complex resources like knowledge graphs to further refine the Knowledge Enhance Layer.
Summary updated