PatchBERT: Just-in-Time, Out-of-Vocabulary Patching

Sangwhan Moon, Naoaki OkazakiEMNLP 2020 · OpenReview

Abstract

Sangwhan Moon, Naoaki Okazaki. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP).

2020.

Introduction

Large-scale multilingual models like BERT face a significant trade-off: supporting dozens of languages within a constrained vocabulary size (typically around 110k tokens). This limitation is particularly acute for Chinese, Japanese, and Korean (CJK) languages, which utilize thousands of unique characters. When these models encounter a character outside their pre-defined vocabulary, they produce an [UNK] (unknown) token, resulting in a total loss of information. While expanding the vocabulary is a possible solution, it increases the model's memory footprint and computational overhead.

PatchBERT introduces a "just-in-time" method to address the out-of-vocabulary (OOV) problem without increasing the model's parameter count. By replacing unknown tokens with "surrogate" tokens already present in the vocabulary, the authors recover lost information during fine-tuning.

Methodology: Surrogate Patching

The core of PatchBERT is Surrogate Patching, a technique where OOV subwords are mapped to existing in-vocabulary tokens. Instead of treating every unknown character as a generic [UNK], the system assigns a unique identifier from the existing vocabulary to each OOV character.

Surrogate Patching mechanism

The authors explore three primary strategies for selecting these surrogates:

  1. Character Distance: This method leverages the structure of Unicode blocks. It calculates the codepoint distance between an OOV character and in-vocabulary characters of the same length. In CJK scripts, adjacent characters often share radicals (semantic similarity) or phonetic components.
  2. Unseen Tokens: The system identifies tokens in the existing vocabulary that do not appear in the specific downstream task and re-purposes them as "blank slates" for OOV characters.
  3. Masked Language Model (MLM) Proposal: The OOV token is replaced with a [MASK] tag. The pre-trained BERT model then predicts the most likely in-vocabulary substitute based on context. The most frequent high-probability prediction is selected as the surrogate.
Unicode similarity in CJK blocks

Experimental Setup

The researchers evaluated PatchBERT on four datasets representing diverse CJK linguistic challenges:

  • NSMC (Naver Sentiment Movie Corpus): A Korean sentiment analysis task with a high OOV rate (30.1%) due to slang and typos.
  • Japanese Twitter Sentiment: A dataset of 22k tweets with a 25.1% OOV rate, primarily driven by emojis and casual language.
  • INEWS: A Chinese news sentiment dataset (20.1% OOV).
  • KorQuAD 1.0: A Korean reading comprehension (QA) dataset with a lower OOV rate (5.9%) based on formal Wikipedia text.

Key Insights

A critical finding is that Information Preservation is often more important than perfect semantic accuracy. Even if a surrogate is not a perfect match, providing a unique identifier allows the model to learn the specific meaning of an OOV token during the fine-tuning process. Furthermore, the Character Distance method proves to be a computationally efficient proxy for similarity in CJK languages due to the organized nature of their Unicode encoding.

Results

PatchBERT consistently outperformed the baseline BERT model across the evaluated tasks. In the Korean NSMC task, the Character Distance method improved accuracy by 1.12%. For Japanese Twitter, the Masked LM approach yielded a 1.76% improvement. Even in the formal Korean QA task (KorQuAD), patching improved the F1 score by 0.69% despite the lower OOV rate. Improvements in the Chinese INEWS dataset were more modest (0.6% to 1.5%), likely because news classification relies heavily on common keywords already present in the vocabulary.

Conclusion and Future Work

PatchBERT demonstrates that "zero-parameter" gains are possible by intelligently re-using existing vocabulary. However, the Character Distance method is script-specific and may not generalize to Latin or Cyrillic alphabets. Additionally, mapping OOV tokens to existing surrogates introduces potential polysemy, which might affect zero-shot performance. Future work could explore the impact of patching on generative tasks, where "de-patching" surrogates back into original characters would be necessary.

Summary updated

BibTeX
@inproceedings{moon2020patchbert,
  author = {Sangwhan Moon and Naoaki Okazaki},
  title = {{PatchBERT: Just-in-Time, Out-of-Vocabulary Patching}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2020},
  url = {https://openreview.net/forum?id=02YhzGLBCrO},
}
Figure