Bridging the Gap between Subword and Character Segmentation in Pretrained Language Models

Shun Kiyono, Sho Takase, Shengzhe Li, Toshinori SatoRanlp 2023

Abstract

Pretraining large-scale language models (PLMs) is computationally expensive. Currently, practitioners often pretrain separate models for subword-level tasks (e.g., sentiment analysis) and character-level tasks (e.g., punctuation restoration). We propose a unified pretraining strategy using BPE-dropout that allows a single model to excel at both granularities. By randomly dropping merge rules during pretraining, the model learns a flexible representation of text. Our results on Japanese benchmarks show that this unified model achieves performance comparable to or better than specialized models while halving the total pretraining cost.

Introduction

The choice of tokenization strategy is a foundational decision in Natural Language Processing. Most modern PLMs utilize subword segmentation, such as Byte Pair Encoding (BPE), to balance vocabulary size and computational efficiency. While effective for many tasks, subword segmentation can obscure character-level information necessary for tasks like spelling correction or punctuation restoration. For instance, in Japanese, a subword tokenizer might group a particle and the start of the next word into a single token, making it impossible for the model to insert a comma between them.

Comparison of previous specialized pretraining vs. the proposed unified approach.

To address this, the current industry standard involves pretraining two distinct models: a Character PLM and a Subword PLM. This "double pretraining" is inefficient, doubling the required GPU hours and environmental impact. We propose replacing these specialized models with a single unified PLM.

Methodology: Unified Segmentation via BPE-dropout

We propose a "unified segmentation" approach that leverages BPE-dropout during the pretraining phase. Unlike standard BPE, which follows deterministic merge rules to combine characters into subwords, BPE-dropout introduces stochasticity.

Illustration of standard BPE vs. BPE-dropout merge rules.

During pretraining, merge rules are ignored with a probability $p$ (we used $p=0.1$). This forces the model to encounter the same text in various states of decomposition—sometimes as full subwords, sometimes as individual characters, and often as a mixture. This exposure builds a model that is inherently robust to different granularities.

Flexible Finetuning

A key advantage of this approach is that the same pretrained weights can be adapted to specific tasks during finetuning by simply adjusting the dropout rate:

  • Subword Tasks: Set $p=0.0$ for deterministic subword input.
  • Character Tasks: Set $p=1.0$ to force the model to treat every character as an individual token.

This transition requires no architectural changes and introduces no additional parameters.

Experimental Results

We pretrained a BERT-base model on a 2020 dump of Japanese Wikipedia and evaluated it against specialized baselines.

Subword-Level Performance (JGLUE)

On the Japanese General Language Understanding Evaluation (JGLUE) benchmark, the unified model performed competitively with standard subword-only BERT. For the JNLI (Natural Language Inference) task, the unified model achieved a score of 88.69, compared to the baseline's 89.43.

Character-Level Performance (Punctuation Restoration)

To test character-level precision, we used a punctuation restoration task on the CC-100 Japanese corpus. The unified model achieved an F1 score of 82.06, outperforming the dedicated character-level model (81.13). This suggests that the subword information learned during unified pretraining provides beneficial context even for character-level predictions.

Conclusion

Our work demonstrates that BPE-dropout is not just a regularization tool but a bridge between linguistic granularities. By unifying subword and character pretraining, we can significantly reduce the computational burden of developing high-performance PLMs without sacrificing accuracy. Future work will explore the applicability of this method to other languages and architectures, such as encoder-decoder (T5) or decoder-only (GPT) models.

Summary updated

BibTeX
@inproceedings{kiyono2023bridging,
  author = {Shun Kiyono and Sho Takase and Shengzhe Li and Toshinori Sato},
  title = {{Bridging the Gap between Subword and Character Segmentation in Pretrained Language Models}},
  booktitle = {Ranlp},
  year = {2023},
}
Figure