Searching for Optimal Subword Tokenization in Cross-domain NER

Ruotian Ma, Yiding Tan, Xin Zhou, Xuanting Chen, Di Liang, Sirui Wang, Wei Wu, Tao Gui, Qi Zhang

TL;DR

X-Piece is a novel subword-level solution for cross-domain Named Entity Recognition (NER) that tackles the "input distribution shift" problem. By re-tokenizing source domain text to align its subword distribution with the target domain using Optimal Transport, it significantly improves model generalization without requiring any labeled data in the target domain.

What Problem Does This Paper Address?

In Unsupervised Domain Adaptation (UDA), the goal is to train a model on a labeled "source" domain (e.g., news articles) and have it perform well on an unlabeled "target" domain (e.g., social media). Most existing methods focus on Domain-Invariant Representation Learning (DIRL), which tries to make high-level neural features look similar across domains.

However, word-level tasks like NER are incredibly sensitive to the raw input distribution. A word like "Madrid" might be a Location in news but part of an Organization ("Real Madrid") in sports. Standard tokenizers (like BERT's WordPiece) are static; they split words the same way regardless of the domain. This creates a discrepancy at the very first layer of the model. The authors argue that aligning these input-level subword distributions is a crucial, yet overlooked, step for successful cross-domain transfer.

Methods

The authors introduce X-Piece, which treats subword tokenization as a dynamic optimization problem rather than a fixed pre-processing step.

  1. Target Distribution Estimation: Since the target domain is unlabeled, they use an entity lexicon (a list of known entities) to create "noisy" distant annotations. While too messy to train on directly, these annotations are sufficient to estimate the target's subword-label distribution.
  2. Optimal Transport (OT) Formulation: They formulate the search for an optimal tokenizer as an OT problem. The goal is to "transport" the mass from the source subword distribution to the word-label distribution while minimizing the cost (defined by the KL-divergence between source and target distributions).
  3. Solving with Sinkhorn: The OT problem is solved efficiently using the Sinkhorn algorithm to find an optimal joint distribution $P(t, w, y)$ (subwords, words, and labels).
  4. Re-tokenization: Finally, they re-tokenize the source domain data by sampling new subword segmentations based on the optimized distribution. This "warps" the source data to look more like the target data at the subword level before it ever hits the transformer layers.
Figure 1: An intuitive overview of the proposed method. As seen, there is a large discrepancy between the source and target word distributions, as well as the default subword distributions. However, after altering the subword segmentation of the source domain, the subword distributions get much closer.

Datasets

The researchers evaluated X-Piece across four major NER benchmarks representing diverse linguistic styles:

  • CoNLL’03: Newswire data (the standard "source" for many experiments).
  • OntoNotes 5.0: A massive dataset with six distinct sub-domains (Broadcast Conversation, Magazine, Web, etc.).
  • Twitter: Informal social media text.
  • Webpages: General website content.
Table showing the statistics of the datasets used.

Key Insights and Ideas

  • Tokenization is a Learnable Parameter: Instead of treating WordPiece as a "black box," X-Piece shows that how we split words can be a powerful lever for domain adaptation.
  • Distribution vs. Labels: A key "aha moment" is that while noisy lexicon labels are bad for direct training, they are excellent for estimating distributions. X-Piece is robust to noise because it only cares about the aggregate statistics, not the correctness of every individual tag.
  • Complementary Nature: Because X-Piece works at the input level, it can be stacked with traditional feature-level methods like DANN (Domain-Adversarial Neural Networks). The paper shows that doing both yields better results than either alone.
  • KL-Divergence as a Compass: The authors found a strong positive correlation between reducing the KL-divergence of subword distributions and the final F1-score gain, validating their optimization objective.

Key Results

  • Consistent Gains: X-Piece consistently outperformed a standard BERT-tagger across almost all domain pairs.
  • Low-Resource Success: The method was particularly effective in low-resource settings. For example, when transferring from Telephone Conversations (TC) to Magazines (MZ) in OntoNotes, it achieved a 4.58% improvement.
  • State-of-the-Art Combinations: Combining X-Piece with DANN achieved the best performance on several benchmarks, such as reaching an F1 of 79.99 on the Webpages dataset when trained on CoNLL'03.
  • Efficiency: Despite the complex OT math, the pre-processing is fast. It takes less than a minute to process ~60k sentences on a standard CPU.
Figure 4: Computational cost of the pre-processing of X-Piece against corpus size. The computational cost is acceptable (up to 58.84s) even with a distribution matrix size of 30.8k × 7.8k.

Limitations and Future Work

  • Lexicon Dependency: The method currently requires an entity lexicon for the target domain. While these are often available, the performance might degrade in extremely niche domains where no such list exists.
  • Task Scope: The study focused exclusively on NER. Future work could explore if optimal subword tokenization benefits other word-level tasks like Part-of-Speech (POS) tagging or Chinese Word Segmentation.
  • Joint vs. Conditional: The authors noted that while optimizing the conditional distribution $P(y|x)$ usually works best, small datasets sometimes benefit more from optimizing the joint distribution $P(x, y)$. Refining when to use which objective remains an open question.

Summary updated

Abstract

Input distribution shift is one of the vital problems in unsupervised domain adaptation (UDA). The most popular UDA approaches focus on domain-invariant representation learning, trying to align the features from different domains into similar feature distributions. However, these approaches ignore the direct alignment of input word distributions between domains, which is a vital factor in word-level classification tasks such as cross-domain NER. In this work, we shed new light on cross-domain NER by introducing a subword-level solution, X-Piece, for input word-level distribution shift in NER. Specifically, we re-tokenize the input words of the source domain to approach the target subword distribution, which is formulated and solved as an optimal transport problem. As this approach focuses on the input level, it can also be combined with previous DIRL methods for further improvement. Experimental results show the effectiveness of the proposed method based on BERT-tagger on four benchmark NER datasets. Also, the proposed method is proved to benefit DIRL methods such as DANN.
BibTeX
@inproceedings{ma2022searching,
  author = {Ruotian Ma and Yiding Tan and Xin Zhou and Xuanting Chen and Di Liang and Sirui Wang and Wei Wu and Tao Gui and Qi Zhang},
  title = {{Searching for Optimal Subword Tokenization in Cross-domain NER}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2022},
  url = {https://openreview.net/forum?id=hPWv0tURf},
}
Figure