Abstract
Input distribution shift is one of the vital problems in unsupervised domain adaptation (UDA). The most popular UDA approaches focus on domain-invariant representation learning (DIRL), trying to align the features from different domains into a similar feature distribution. However, these approaches ignore the direct alignment of input word distributions between domains, which is a vital factor in word-level classification tasks such as cross-domain NER. In this work, we shed new light on cross-domain NER by introducing a subword-level solution, X-Piece, for input word-level distribution shift in NER. Specifically, we re-tokenize the input words of the source domain to approach the target subword distribution, which is formulated and solved as an optimal transport problem. As this approach focuses on the input level, it can also be combined with previous DIRL methods for further improvement. Experimental results show the effectiveness of the proposed method based on BERT-tagger on four benchmark NER datasets. Also, the proposed method is proved to benefit DIRL methods such as DANN.
Introduction: The Challenge of Input Distribution Shift
Cross-domain Named Entity Recognition (NER) often fails because the distribution of words and their meanings shifts between domains. For instance, "Madrid" might typically refer to a city in news articles but frequently denotes a sports club in social media contexts. While Unsupervised Domain Adaptation (UDA) typically employs Domain-Invariant Representation Learning (DIRL) to align high-level features, these methods often overlook the fundamental issue of input distribution shift.
Standard tokenizers, such as BERT’s WordPiece, are static and segment words identically regardless of the domain. This is problematic for word-level tasks like NER because the relationship between subwords and entity labels changes across domains. If a model encounters a subword in the source domain rarely associated with a specific entity type in the target domain, misclassification is likely.
Methodology: The X-Piece Framework
The authors propose X-Piece, shifting the focus from feature-level alignment to input-level alignment. The core innovation lies in re-tokenizing source domain words so their subword distribution matches that of the target domain.
1. Target Distribution Approximation
Since target labels are unknown in UDA, X-Piece utilizes an existing entity lexicon to generate "noisy" distant annotations on the target data. This provides a rough but functional estimate of the target subword-label distribution.
2. Optimal Transport (OT) Formulation
The re-tokenization process is framed as an Optimal Transport problem. The goal is to "transport" the mass of the source subword distribution to match the target distribution while minimizing a cost function based on KL-divergence.
3. Optimization and Sampling
The discrete optimization problem is solved efficiently using the Sinkhorn algorithm to find an optimal mapping matrix. Instead of traditional greedy "longest-match" tokenization, source data is re-tokenized by sampling segmentations from this optimized distribution, ensuring the source data "looks" more like the target domain during training.
Experimental Setup
X-Piece was evaluated on several benchmark NER datasets:
- CoNLL’03: News domain (Source).
- OntoNotes 5.0: Diverse sub-domains including Broadcast Conversation (BC), News (BN), Magazine (MZ), Newswire (NW), Telephone Conversation (TC), and Web (WB).
- Twitter: Social media domain.
- Webpages: General website content.
Key Results and Insights
- Consistent Performance Gains: X-Piece consistently outperformed the standard BERT-tagger across nearly all domain pairs. In OntoNotes cross-domain settings, it achieved up to a 4.58% improvement in F1-score (e.g., TC to MZ).
- Synergy with Feature-based Methods: As an input-level approach, X-Piece is highly complementary to DIRL methods like DANN. Combining them yielded state-of-the-art results, such as 79.54 F1 on CoNLL’03 to OntoNotes.
- Robustness and Efficiency: Despite using noisy lexicon-based labels, the distributional estimation proved sufficient for guiding effective re-tokenization. Furthermore, the pre-processing is computationally efficient, taking under 60 seconds for a 60k-sentence corpus.
- Correlation with KL-Divergence: Empirical evidence showed that reducing the KL-divergence between source and target subword distributions directly correlates with increased target domain F1-scores.
Conclusion
X-Piece demonstrates that treating tokenization as an optimizable process rather than a fixed pre-processing step can significantly mitigate vocabulary-level domain gaps. While currently dependent on target-domain lexicons, it opens new avenues for input-level adaptation in NLP. Future work may explore dynamic, "on-the-fly" tokenization adjustments during training and application to other sequence labeling tasks.
Summary updated