TL;DR
CharacterBERT replaces BERT’s standard subword (WordPiece) tokenization with a character-level CNN module, enabling word-level, open-vocabulary representations within a Transformer architecture. This approach significantly improves performance and robustness in specialized domains like medicine, where general-purpose subword vocabularies often struggle with complex terminology and misspellings.
What Problem Does This Paper Address?
Most modern Transformer models, like BERT, rely on WordPiece tokenization. This system breaks words into smaller sub-units (e.g., "playing" becomes "play" + "##ing") based on a predefined vocabulary. While this helps handle rare words, it has significant drawbacks:
- Domain Mismatch: A WordPiece vocabulary built on Wikipedia doesn't translate well to specialized fields. For example, the medical term "choledocholithiasis" might be shattered into nonsensical fragments by a general-domain tokenizer.
- Conceptual Complexity: Shifting from word-level to subword-level makes models harder to interpret and less convenient for tasks that are inherently word-based.
- Brittleness: Subword models are notoriously sensitive to noise. A single typo can change a word's tokenization entirely, leading to wildly different representations.
The authors demonstrate this mismatch in Figure 1, showing that a general-domain vocabulary splits medical terms much more aggressively than a domain-specific one, potentially inducing bias in downstream tasks.
Methods
The core innovation is CharacterBERT, which swaps BERT's embedding layer for a Character-CNN module (inspired by ELMo).
Instead of looking up a subword in a table, the model:
- Converts each word into a sequence of characters.
- Passes these through character embeddings and multiple 1D-CNNs with varying filter sizes.
- Uses max-pooling and Highway layers to produce a single, fixed-dimension vector (768-d) for the entire word.
This architecture, contrasted with vanilla BERT in Figure 2, allows the model to remain entirely word-level while still being "open-vocabulary"—it can generate a representation for any string of characters it encounters.
For pre-training, the authors use the standard Masked Language Modeling (MLM) and Next Sentence Prediction (NSP) tasks, but with a twist: in MLM, they mask and predict entire words rather than subwords.
Datasets
The authors pre-trained four model variants using:
- General Domain: English Wikipedia and OpenWebText.
- Medical Domain: MIMIC-III clinical notes and PMC OA biomedical abstracts.
Evaluation was conducted on five specialized medical benchmarks:
- i2b2/VA 2010: Clinical concept extraction (NER).
- MEDNLI: Clinical natural language inference.
- ChemProt & DDI: Biomedical relation classification (chemical-protein and drug-drug interactions).
- ClinicalSTS: Clinical sentence similarity.
Key Insights and Ideas
- Word-Level Simplicity: By returning to word-level representations, the model avoids the "fragmentation" problem of subwords, making it more intuitive for practitioners.
- Open-Vocabulary Robustness: Because it builds representations from characters, it never encounters an "out-of-vocabulary" word. It can gracefully handle medical jargon and technical terms it hasn't seen during training.
- Efficiency at Inference: While the Character-CNN is computationally heavier than a simple lookup table, CharacterBERT can actually be faster at inference for certain tasks. This is because it processes fewer tokens (words vs. subwords), leading to shorter input sequences for the Transformer layers.
Key Results
- Superior Performance: CharacterBERT outperformed vanilla BERT on nearly all medical tasks. Statistical significance tests (ASO) confirmed that the medical version of CharacterBERT was the overall best-performing model across the board.
- Noise Tolerance: CharacterBERT is significantly more robust to typos. As shown in Figure 7, when 40% noise (misspellings) was added to the MEDNLI test set, CharacterBERT maintained a ~5 F1 point lead over BERT.
- Parameter Efficiency: Despite the "complex" CNN module, CharacterBERT actually has fewer total parameters (~104.6M) than BERT (~109.5M) because it eliminates the massive WordPiece embedding matrix.
Limitations and Future Work
The primary drawback is pre-training speed. CharacterBERT is roughly 2x slower to pre-train than BERT. This is partly due to the CNN's complexity and partly because it cannot use the "weight-sharing" trick between the input embeddings and the output layer that speeds up standard BERT.
Future work suggested by the authors includes:
- Optimizing pre-training speed using techniques like Noise Contrastive Estimation.
- Applying the Character-CNN architecture to more recent Transformer variants like RoBERTa or ALBERT.
- Testing the approach on non-English languages and other specialized domains (e.g., legal or financial).
Summary updated