Abstract
Standard transformer models rely on fixed subword vocabularies, which struggle with out-of-vocabulary words and domain shifts. This paper introduces XRAYEMB, a lightweight retrofitting method that augments existing pre-trained language models with character-level encoders and decoders. By injecting character-level awareness into already pre-trained models, XRAYEMB improves performance on word-level tasks like Named Entity Recognition (NER) and novel word classification without the computational cost of training from scratch.
1. Introduction: The Subword Shattering Problem
Modern pre-trained language models (e.g., BERT, GPT-2, RoBERTa) use subword tokenization (WordPiece, BPE) with a frozen vocabulary. When these models encounter text from new domains, such as noisy user-generated tweets, they often "shatter" unknown words into many meaningless subword tokens. For example, the hashtag #4thofjuly might be split into #, 4th, ##of, ##ju, ##ly. This fragmentation destroys the word's contiguous representation and artificially inflates sequence lengths.
As shown in Table 1, noisy datasets like Twitter and Emoji Prediction exhibit significantly higher "Multitoks" percentages and "Token mass increase" compared to Wikipedia baselines, highlighting the failure of standard tokenizers on informal text.
2. Methods: The XRAYEMB Architecture
XRAYEMB is a plug-and-play architecture designed to retrofit existing transformers. It consists of three main components:
- XR-ENC (Encoder): A lightweight character-level CNN that computes a single vector representation for a word based on its characters. This vector dynamically replaces standard subword embeddings for specific tokens (e.g., multi-token or rare words) based on a defined policy.
- XR-DEC (Decoder): An LSTM-based generative component that decodes the transformer's contextualized output back into a character sequence.
- Second Pre-training (2PT): A short, secondary pre-training phase (one epoch on ~725MB of text) to "acclimatize" the base transformer to the new character-based embeddings.
The model utilizes "Cycle Dependency Loops" during 2PT as a regularization technique to ensure the character and vector spaces remain tightly aligned (Character $\rightarrow$ XR-ENC $\rightarrow$ XR-DEC $\rightarrow$ Character).
3. Results and Discussion
The authors evaluated XRAYEMB on several tasks, including Emoji Prediction, Twitter NER, and NYTWIT (novel word classification).
- Word-level Tasks: Character-level augmentation shines on word-level tasks where internal word structure is crucial. On the NYTWIT task, BERT's F1 score jumped from 45.21 to 53.62.
- NER Performance: BERT's F1 on Twitter NER improved from 29.06 to 34.09, and GPT-2 improved from 30.25 to 36.27.
- RoBERTa Resilience: RoBERTa's heavily optimized pre-training makes it incredibly robust, capturing fine-grained word info better than BERT or GPT-2, resulting in smaller gains from XRAYEMB.
- Efficiency: The 2PT phase is highly efficient, requiring only a single epoch to stabilize loss metrics, proving that retrofitting is vastly cheaper than training character-aware models from scratch.
4. Conclusion
XRAYEMB provides a cost-effective way to inject character-level awareness into pre-trained token-based models, significantly improving robustness to domain shift and rare words in word-level tasks. Future work will explore extensions to morphologically rich languages and the use of the XR-DEC module in generative downstream tasks.
Summary updated