Abstract
Multilingual sequence labeling is a task of predicting label sequences using a single unified model for multiple languages. Compared with relying on multiple monolingual models, using a multilingual model has the benefit of a smaller model size, easier in online serving, and generalizability to low-resource languages. However, current multilingual models still underperform individual monolingual models significantly due to model capacity limitations. In this paper, we propose to reduce the gap between monolingual models and the unified multilingual model by distilling the structural knowledge of several monolingual models (teachers) to the unified multilingual model (student). We propose two novel KD methods based on structure-level information: (1) approximately minimizes the distance between the student’s and the teachers’ structure-level probability distributions, (2) aggregates the structure-level knowledge to local distributions and minimizes the distance between two local probability distributions. Our experiments on 4 multilingual tasks with 25 datasets show that our approaches outperform several strong baselines and have stronger zero-shot generalizability than both the baseline model and teacher models.
Introduction
In Natural Language Processing (NLP), practitioners often face a trade-off between accuracy and scalability. Monolingual models, often trained on a single language with specialized embeddings like "Flair," achieve high accuracy but are difficult to maintain at scale. Conversely, multilingual models like M-BERT offer a unified solution for hundreds of languages but typically lag in performance due to limited model capacity and the complexity of cross-lingual balancing.
A primary reason for this performance gap is that standard Knowledge Distillation (KD) techniques—where a student model mimics a teacher—usually operate at the token level. While effective for simple classification, token-level KD ignores the global dependencies inherent in sequence labeling tasks (e.g., NER or POS tagging), where the label of one word depends heavily on its neighbors. This research introduces Structure-Level Knowledge Distillation to bridge this gap by transferring the "structural intuition" of expert monolingual teachers into a single multilingual student.
Methodology
The authors propose two novel structure-level distillation methods to train a BiLSTM-CRF student model using M-BERT embeddings:
- Top-K Distillation: A Conditional Random Field (CRF) predicts a probability distribution over an exponentially large number of possible label sequences. Since the student cannot mimic the entire distribution, this method uses a modified Viterbi algorithm to identify the $k$-best sequences from the teacher. The student then attempts to match the teacher's distribution over this manageable subset of paths.
- Posterior Distillation: This method utilizes the forward-backward algorithm to calculate the marginal probability of each label at every position, conditioned on the entire global sequence structure. These "posteriors" encapsulate the structural constraints learned by the teacher, providing a more mathematically exact distillation signal than Top-K approximations.
The student model learns from both human-labeled "gold" data and the structural "pseudo-labels" generated by monolingual teachers, who utilize superior features like Flair or fastText embeddings.
Experimental Results
The approach was evaluated across 25 datasets spanning four tasks: CoNLL NER, WikiAnn NER, Universal Dependencies (UD) POS tagging, and SemEval 2016 Aspect Extraction.
Performance Gains
The "Posterior" distillation method consistently outperformed the multilingual baseline in 21 out of 25 datasets. On the CoNLL NER task, the Posterior student achieved an average F1 score of 87.72, significantly narrowing the gap to the expert teachers (89.38) compared to the standard baseline (87.36).
Key Insights
- Structure Trumps Tokens: Distilling the posterior distribution, which accounts for neighboring labels, allows the student to learn the underlying "logic" of the sequence rather than isolated classifications.
- Zero-Shot Generalizability: Surprisingly, the distilled student often outperformed both the baseline and the teachers on "zero-shot" tasks—languages the teachers were never trained on. This suggests that structural distillation helps the model capture more universal, language-agnostic patterns.
- Feature Absorption: The student model can effectively "absorb" the benefits of advanced teacher features (like Flair embeddings) even if the student's own architecture is simpler.
Limitations and Future Work
While effective, structure-level distillation introduces computational overhead, with training times increasing by approximately 1.45x to 1.63x over standard baselines. Additionally, storing teacher structural predictions requires significant CPU memory. Future work will focus on optimizing the fine-tuning process, as simultaneously updating M-BERT and teacher embeddings remains computationally prohibitive for large-scale multilingual applications.
Summary updated