CharacterBERT: Reconciling ELMo and BERT for Word-Level Open-Vocabulary Representations From Characters

Hicham El Boukkouri, Olivier Ferret, Thomas Lavergne, Hiroshi Noji, Pierre Zweigenbaum, Junichi Tsujii

TL;DR

CharacterBERT replaces BERT’s standard subword (WordPiece) tokenization with a character-level CNN module, enabling word-level, open-vocabulary representations within a Transformer architecture. This approach significantly improves performance and robustness in specialized domains like medicine, where general-purpose subword vocabularies often struggle with complex terminology and misspellings.

What Problem Does This Paper Address?

Most modern Transformer models, like BERT, rely on WordPiece tokenization. This system breaks words into smaller sub-units (e.g., "playing" becomes "play" + "##ing") based on a predefined vocabulary. While this helps handle rare words, it has significant drawbacks:

  1. Domain Mismatch: A WordPiece vocabulary built on Wikipedia doesn't translate well to specialized fields. For example, the medical term "choledocholithiasis" might be shattered into nonsensical fragments by a general-domain tokenizer.
  2. Conceptual Complexity: Shifting from word-level to subword-level makes models harder to interpret and less convenient for tasks that are inherently word-based.
  3. Brittleness: Subword models are notoriously sensitive to noise. A single typo can change a word's tokenization entirely, leading to wildly different representations.

The authors demonstrate this mismatch in Figure 1, showing that a general-domain vocabulary splits medical terms much more aggressively than a domain-specific one, potentially inducing bias in downstream tasks.

Figure 1: Comparison of the tokenization of a medical corpus by vocabularies from different domains.

Methods

The core innovation is CharacterBERT, which swaps BERT's embedding layer for a Character-CNN module (inspired by ELMo).

Instead of looking up a subword in a table, the model:

  1. Converts each word into a sequence of characters.
  2. Passes these through character embeddings and multiple 1D-CNNs with varying filter sizes.
  3. Uses max-pooling and Highway layers to produce a single, fixed-dimension vector (768-d) for the entire word.

This architecture, contrasted with vanilla BERT in Figure 2, allows the model to remain entirely word-level while still being "open-vocabulary"—it can generate a representation for any string of characters it encounters.

Comparison of the context-independent representation systems in BERT and CharacterBERT.

For pre-training, the authors use the standard Masked Language Modeling (MLM) and Next Sentence Prediction (NSP) tasks, but with a twist: in MLM, they mask and predict entire words rather than subwords.

Datasets

The authors pre-trained four model variants using:

  • General Domain: English Wikipedia and OpenWebText.
  • Medical Domain: MIMIC-III clinical notes and PMC OA biomedical abstracts.

Evaluation was conducted on five specialized medical benchmarks:

  • i2b2/VA 2010: Clinical concept extraction (NER).
  • MEDNLI: Clinical natural language inference.
  • ChemProt & DDI: Biomedical relation classification (chemical-protein and drug-drug interactions).
  • ClinicalSTS: Clinical sentence similarity.

Key Insights and Ideas

  • Word-Level Simplicity: By returning to word-level representations, the model avoids the "fragmentation" problem of subwords, making it more intuitive for practitioners.
  • Open-Vocabulary Robustness: Because it builds representations from characters, it never encounters an "out-of-vocabulary" word. It can gracefully handle medical jargon and technical terms it hasn't seen during training.
  • Efficiency at Inference: While the Character-CNN is computationally heavier than a simple lookup table, CharacterBERT can actually be faster at inference for certain tasks. This is because it processes fewer tokens (words vs. subwords), leading to shorter input sequences for the Transformer layers.

Key Results

  • Superior Performance: CharacterBERT outperformed vanilla BERT on nearly all medical tasks. Statistical significance tests (ASO) confirmed that the medical version of CharacterBERT was the overall best-performing model across the board.
  • Noise Tolerance: CharacterBERT is significantly more robust to typos. As shown in Figure 7, when 40% noise (misspellings) was added to the MEDNLI test set, CharacterBERT maintained a ~5 F1 point lead over BERT.
  • Parameter Efficiency: Despite the "complex" CNN module, CharacterBERT actually has fewer total parameters (~104.6M) than BERT (~109.5M) because it eliminates the massive WordPiece embedding matrix.

Limitations and Future Work

The primary drawback is pre-training speed. CharacterBERT is roughly 2x slower to pre-train than BERT. This is partly due to the CNN's complexity and partly because it cannot use the "weight-sharing" trick between the input embeddings and the output layer that speeds up standard BERT.

Future work suggested by the authors includes:

  • Optimizing pre-training speed using techniques like Noise Contrastive Estimation.
  • Applying the Character-CNN architecture to more recent Transformer variants like RoBERTa or ALBERT.
  • Testing the approach on non-English languages and other specialized domains (e.g., legal or financial).

Summary updated

Abstract

Due to the compelling improvements brought by BERT, many recent representation models adopted the Transformer architecture as their main building block, consequently inheriting the wordpiece tokenization system despite it not being intrinsically linked to the notion of Transformers. While this system is thought to achieve a good balance between the flexibility of characters and the efficiency of full words, using predefined wordpiece vocabularies from the general domain is not always suitable, especially when building models for specialized domains (e.g., the medical domain). Moreover, adopting a wordpiece tokenization shifts the focus from the word level to the subword level, making the models conceptually more complex and arguably less convenient in practice. For these reasons, we propose CharacterBERT, a new variant of BERT that drops the wordpiece system altogether and uses a Character-CNN module instead to represent entire words by consulting their characters. We show that this new model improves the performance of BERT on a variety of medical domain tasks while at the same time producing robust, word-level and open-vocabulary representations.
BibTeX
@inproceedings{boukkouri2020characterbert,
  author = {Hicham El Boukkouri and Olivier Ferret and Thomas Lavergne and Hiroshi Noji and Pierre Zweigenbaum and Junichi Tsujii},
  title = {{CharacterBERT: Reconciling ELMo and BERT for Word-Level Open-Vocabulary Representations From Characters}},
  booktitle = {International Conference on Computational Linguistics},
  year = {2020},
}
Figure