Abstract
Multilingual sequence labeling is a task of predicting label sequences using a single unified model for multiple languages. Compared with relying on multiple monolingual models, using a multilingual model has the benefit of a smaller model size, easier in online serving, and generalizability to low-resource languages. However, current multilingual models still underperform individual monolingual models significantly due to model capacity limitations. In this paper, we propose to reduce the gap between monolingual models and the unified multilingual model by distilling the structural knowledge of several monolingual models (teachers) to the unified multilingual model (student). We propose two novel KD methods based on structure-level information: (1) approximately minimizes the distance between the student's and the teachers' structure level probability distributions, (2) aggregates the structure-level knowledge to local distributions and minimizes the distance between two local probability distributions. Our experiments on 4 multilingual tasks with 25 datasets show that our approaches outperform several strong baselines and have stronger zero-shot generalizability than both the baseline model and teacher models.
Introduction
In Natural Language Processing (NLP), sequence labeling is a fundamental task used for Named Entity Recognition (NER) and Part-of-Speech (POS) tagging. Traditionally, specialized models are built for each language. However, maintaining thousands of models for every language is resource-intensive. The alternative—a unified multilingual model—is efficient and beneficial for low-resource languages but often suffers from a "capacity gap," underperforming specialized monolingual counterparts. This gap arises because a single model struggles to capture the unique structural nuances of many languages simultaneously. Existing Knowledge Distillation (KD) techniques typically focus on token-level information, ignoring the global structural dependencies within a sentence.
Proposed Methods: Structure-Level Distillation
To address these limitations, we propose moving beyond token-level distillation to Structure-Level Knowledge Distillation. Our framework utilizes a BiLSTM-CRF architecture, where the Conditional Random Field (CRF) layer models the relationships between neighboring labels.
We introduce two novel KD approaches:
- Top-K Distillation: The student model mimics the teacher’s probability distribution over the $k$ most likely label sequences. A modified Viterbi algorithm is used to find these top-k candidates, making the exponentially large sequence space computationally tractable.
- Posterior Distillation: This approach aggregates global structural knowledge into local "posterior" distributions (marginal probabilities) for each token. By employing the forward-backward algorithm, we calculate these probabilities exactly, allowing the student to learn the structural influence of the entire sentence on every word.
Experimental Evaluation
We evaluated our methods across 4 tasks using 25 datasets, including CoNLL NER (English, Dutch, Spanish, German), WikiAnn NER (8 diverse languages), Universal Dependencies for POS tagging, and SemEval 2016 for Aspect Extraction.
The teacher models utilized high-end, language-specific embeddings (e.g., Flair, fastText). Despite the student model using standard multilingual embeddings, it effectively "absorbed" performance gains through structural distillation.
Key Results and Insights
Our experiments demonstrate that distilling the CRF layer's logic is essential. It allows the student to learn why certain sequences are preferred, rather than just predicting individual labels.
As shown in the results table, the Posterior Distillation method consistently outperformed the multilingual baseline. On the CoNLL NER task, it achieved an average F1 score of 87.72, significantly closing the gap with the expert teachers. Posterior distillation generally proved superior to Top-K distillation as it provides a mathematically exact representation of structural knowledge without the bias of selecting only the top sequences.
Furthermore, the distilled student model exhibited superior zero-shot transfer. In tests training on CoNLL and testing on WikiAnn, the Posterior model outperformed the baseline on 24 out of 28 languages, sometimes improving F1 scores by over 30 points compared to teacher models in languages like Arabic and Bulgarian. This suggests that learning structural patterns from multiple teachers fosters a more universal linguistic understanding.
Conclusion and Future Work
While structure-level KD provides significant performance gains, it introduces a computational trade-off, with training times increasing by approximately 1.45x to 1.63x due to the forward-backward algorithm. Future research could explore fine-tuning underlying models like M-BERT during distillation and developing more memory-efficient ways to handle teacher distributions for larger-scale deployments.
Summary updated