Abstract
Massively Multilingual Transformers (MMTs) often struggle with cross-lingual transfer for Named Entity Recognition (NER) when moving from high-resource source languages to low-resource target languages. This paper identifies "decontextualization"—an over-reliance on individual token features like capitalization—as a primary cause of transfer failure. We propose SLICER (Sliced Fine-Tuning for NER), a regularization technique that divides token representations into independent slices during fine-tuning. This architectural constraint forces the model to encode more robust, universal contextual information. Our results demonstrate that SLICER significantly outperforms standard fine-tuning, particularly on linguistically distant African languages, achieving performance gains of up to 8 F1 points.
Introduction: The Decontextualization Problem
While models like mBERT and XLM-R are the workhorses of modern NLP, their transfer capabilities often hit a wall in low-resource settings. The authors identify a specific culprit: during standard NER fine-tuning, the model's higher layers start paying more attention to the token itself and less to its surroundings. In English, the model learns "shortcuts"—for example, if a word is capitalized, it is likely an entity. However, many low-resource languages do not use capitalization or have entirely different morphological markers. When a model relies on these source-specific shortcuts, it fails to recognize entities in target languages that lack them.
Methodology: SLICER
To combat this "lazy" learning, the authors introduce SLICER. The core idea is to restrict the capacity of the classifier to prevent it from memorizing complex, non-contextual correlations within the token vector.
As illustrated in the figure above, SLICER operates through three main steps:
- Slicing: The high-dimensional token vector (e.g., 768 dimensions) is divided into $d/h$ disjoint slices of size $h$ (where $h$ is typically 1, 2, or 8).
- Independent Classification: Each slice is treated as an independent representation and passed through its own subset of classification parameters.
- Joint Loss: The model computes a loss for every single slice and averages them.
By forcing each small slice to be "self-sufficient" for classification, the Transformer layers are coerced into embedding more meaningful, contextual information into every part of the representation.
Experimental Setup
The researchers evaluated SLICER using English as the primary source language across two major benchmarks:
- WikiANN: A silver-standard dataset covering 23 target languages.
- MasakhaNER: A high-quality, human-annotated dataset focusing on 10 African languages (e.g., Amharic, Igbo, Yoruba, Swahili).
Key Results and Analysis
SLICER consistently outperformed standard fine-tuning, with the performance gap growing as the target language became more linguistically distant from the source.
Key insights from the evaluation include:
- Performance Gains: SLICER achieved impressive gains of approximately 8 F1 points over standard fine-tuning on the MasakhaNER African language benchmark.
- Reduced Decontextualization: Analysis of attention maps confirmed that SLICER successfully reduced the "attention-to-self" proportion in higher layers, proving that the model maintained stronger contextualization.
- Robustness: The method proved effective across different learning rates and slice sizes, suggesting it is a stable and easy-to-implement enhancement for NER tasks.
Conclusion and Future Work
SLICER demonstrates that restricting classifier capacity can act as a powerful regularizer for cross-lingual transfer by prioritizing universal context over local token features. While currently optimized for NER, future work will investigate why certain tasks decontextualize more than others and whether the "slicing" logic can be adapted for broader multilingual representation learning beyond token-level classification.
Summary updated