Learning to Look Inside: Augmenting Token-Based Encoders with Character-Level Information

Yuval Pinter, Amanda Stent, Mark Dredze, Jacob EisensteinarXiv 2021 · arXiv

Abstract

Standard transformer models rely on fixed subword vocabularies, which struggle with out-of-vocabulary words and domain shifts. This paper introduces XRAYEMB, a lightweight retrofitting method that augments existing pre-trained language models with character-level encoders and decoders. By injecting character-level awareness into already pre-trained models, XRAYEMB improves performance on word-level tasks like Named Entity Recognition (NER) and novel word classification without the computational cost of training from scratch.

1. Introduction: The Subword Shattering Problem

Modern pre-trained language models (e.g., BERT, GPT-2, RoBERTa) use subword tokenization (WordPiece, BPE) with a frozen vocabulary. When these models encounter text from new domains, such as noisy user-generated tweets, they often "shatter" unknown words into many meaningless subword tokens. For example, the hashtag #4thofjuly might be split into #, 4th, ##of, ##ju, ##ly. This fragmentation destroys the word's contiguous representation and artificially inflates sequence lengths.

Table 1: Dataset statistics illustrating the tokenization overhead (Multitoks and Token mass increase) across various domains compared to Wikipedia.

As shown in Table 1, noisy datasets like Twitter and Emoji Prediction exhibit significantly higher "Multitoks" percentages and "Token mass increase" compared to Wikipedia baselines, highlighting the failure of standard tokenizers on informal text.

2. Methods: The XRAYEMB Architecture

XRAYEMB is a plug-and-play architecture designed to retrofit existing transformers. It consists of three main components:

  • XR-ENC (Encoder): A lightweight character-level CNN that computes a single vector representation for a word based on its characters. This vector dynamically replaces standard subword embeddings for specific tokens (e.g., multi-token or rare words) based on a defined policy.
  • XR-DEC (Decoder): An LSTM-based generative component that decodes the transformer's contextualized output back into a character sequence.
  • Second Pre-training (2PT): A short, secondary pre-training phase (one epoch on ~725MB of text) to "acclimatize" the base transformer to the new character-based embeddings.
Figure 1: The XRAYEMB architecture, showing the integration of the XR-ENC and XR-DEC modules with a base transformer (M).

The model utilizes "Cycle Dependency Loops" during 2PT as a regularization technique to ensure the character and vector spaces remain tightly aligned (Character $\rightarrow$ XR-ENC $\rightarrow$ XR-DEC $\rightarrow$ Character).

3. Results and Discussion

The authors evaluated XRAYEMB on several tasks, including Emoji Prediction, Twitter NER, and NYTWIT (novel word classification).

  • Word-level Tasks: Character-level augmentation shines on word-level tasks where internal word structure is crucial. On the NYTWIT task, BERT's F1 score jumped from 45.21 to 53.62.
  • NER Performance: BERT's F1 on Twitter NER improved from 29.06 to 34.09, and GPT-2 improved from 30.25 to 36.27.
  • RoBERTa Resilience: RoBERTa's heavily optimized pre-training makes it incredibly robust, capturing fine-grained word info better than BERT or GPT-2, resulting in smaller gains from XRAYEMB.
  • Efficiency: The 2PT phase is highly efficient, requiring only a single epoch to stabilize loss metrics, proving that retrofitting is vastly cheaper than training character-aware models from scratch.

4. Conclusion

XRAYEMB provides a cost-effective way to inject character-level awareness into pre-trained token-based models, significantly improving robustness to domain shift and rare words in word-level tasks. Future work will explore extensions to morphologically rich languages and the use of the XR-DEC module in generative downstream tasks.

Summary updated

Abstract

Commonly-used transformer language models depend on a tokenization schema which sets an unchangeable subword vocabulary prior to pre-training, destined to be applied to all downstream tasks regardless of domain shift, novel word formations, or other sources of vocabulary mismatch. Recent work has shown that "token-free" models can be trained directly on characters or bytes, but training these models from scratch requires substantial computational resources, and this implies discarding the many domain-specific models that were trained on tokens. In this paper, we present XRayEmb, a method for retrofitting existing token-based models with character-level information. XRayEmb is composed of a character-level "encoder" that computes vector representations of character sequences, and a generative component that decodes from the internal representation to a character sequence. We show that incorporating XRayEmb's learned vectors into sequences of pre-trained token embeddings helps performance on both autoregressive and masked pre-trained transformer architectures and on both sequence-level and sequence tagging tasks, particularly on non-standard English text.
BibTeX
@misc{pinter2021learning,
  author = {Yuval Pinter and Amanda Stent and Mark Dredze and Jacob Eisenstein},
  title = {{Learning to Look Inside: Augmenting Token-Based Encoders with Character-Level Information}},
  year = {2021},
  eprint = {2108.00391},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2108.00391},
}
Figure