Abstract
Yanbo J. Wang, Sheng Chen, Hengxing Cai, Wei Wei, Kuo Yan, Zhe Sun, Hui Qin, Yuming Li, Xiaochen Cai. Proceedings of the Towards Semi-Supervised and Reinforced Task-Oriented Dialog Systems (SereTOD).
2022.
Introduction
Task-Oriented Dialogue (TOD) systems are designed to help users accomplish specific goals through natural language interaction. A critical component of these systems is Spoken Language Understanding (SLU), which extracts structured information from user utterances. However, real-world dialogue transcripts, such as those from customer service centers, present significant challenges: nested entities, frequent coreferences (e.g., "that package"), long slot values, and a gap between training and real-world data distributions.
This paper presents the winning solution for Track 1 of the SereTOD 2022 Workshop challenge. The authors propose a robust pipeline centered on the GlobalPointer architecture, enhanced by multi-model fusion and context-aware masking, to handle the complexities of multi-turn Chinese customer service dialogues.
Methodology
GlobalPointer for Span-based NER
The core of the extraction engine is GlobalPointer, a span-based method for Named Entity Recognition (NER). Traditional sequence labeling (like BIO tagging) often struggles with nested entities. GlobalPointer addresses this by treating every possible start-end position pair as a potential entity. These pairs are represented in an upper triangular matrix, where each cell indicates the probability of a specific span being an entity of a certain type.
Global Context Integration
To resolve coreferences and utilize information from previous turns, the authors concatenate the dialogue history to the current input. To ensure the model focuses on the current turn while referencing the past, they implement a dual-masking strategy. An attention_mask allows the model to attend to the entire history, while a valid_mask restricts loss calculation to the current dialogue turn. This approach provided a 2-percentage-point improvement in performance.
Multi-Model Fusion and Engineering Refinements
The final system employs an ensemble of five transformer backbones: Roformer, DeBERTa, RoBERTa, MacBERT, and NEZHA. Diversity was further increased by using different maximum token lengths (256, 280, 384) across the models.
Additional refinements include:
- Character-Level Tokenization: Splitting text into single characters before BERT tokenization to prevent errors with specialized terms or pinyin.
- MLM Pre-training: Domain adaptation through Masked Language Model pre-training on the specific dialogue dataset.
- Matrix Truncation: Optimizing memory usage during fusion by truncating probability matrices based on maximum expected entity lengths.
Dataset and Results
The system was evaluated on the MobileCS dataset, containing approximately 100,000 real-world Chinese customer service dialogues. The proposed approach secured first place in the SereTOD 2022 Track 1 challenge, achieving an Average F1 of 58.37. The ensemble strategy significantly outperformed any single model, with the Roformer-based model being the strongest individual performer.
| Metric | Score (F1) |
|---|---|
| Entity Extraction | 60.73 |
| Entity-Slot-Value Triple | 56.00 |
| Average | 58.37 |
Conclusion
The success of this approach highlights the effectiveness of span-based NER for complex, nested information extraction tasks. By combining the structural advantages of GlobalPointer with robust context integration and a diverse model ensemble, the authors demonstrated a state-of-the-art solution for processing messy, real-world dialogue transcripts. Future work may focus on reducing the computational overhead of the multi-model ensemble for real-time applications.
Summary updated