Introduction
State-of-the-art NLP models like BERT and ELMo are typically pretrained on "canonical" text: Wikipedia, modern news, and contemporary books. However, language is diverse. When these models are applied to "non-standard" domains—such as Early Modern English (EME) or social media—their performance often degrades significantly. This paper explores AdaptaBERT, a method to adapt pretrained models to these challenging domains using only unlabeled text.
The Challenge of Domain Shift
The difficulty in applying modern NLP models to historical or informal text is twofold:
- Domain Shift: Historical texts, like those from the 17th century, feature inconsistent orthography (e.g., spelling "him" as "hym") and archaic syntax (e.g., "-th" suffixes like "hath").
- Data Scarcity: While labeled data is abundant for modern news (the "source domain"), it is extremely scarce for niche or historical "target domains."
The authors investigate whether the vast amounts of available unlabeled text in these target domains can bridge this gap.
Methodology: AdaptaBERT
The proposed strategy, domain-adaptive fine-tuning, introduces an intermediate unsupervised step between general pretraining and task-specific training:
- Domain Tuning (Unsupervised): The standard pretrained BERT model undergoes continued training using the Masked Language Modeling (MLM) objective. This phase utilizes unlabeled text from the target domain (e.g., 17th-century letters) mixed with source domain text, allowing the model to learn the vocabulary and distributional patterns of the new domain.
- Task Tuning (Supervised): The "domain-aware" model is then fine-tuned on the target task (e.g., Part-of-Speech tagging) using labeled data from the source domain (modern news).
Experimental Results
Early Modern English Tagging
The method was evaluated on the Penn-Helsinki Corpus of Early Modern English (PPCEME). AdaptaBERT demonstrated a significant performance boost, particularly for out-of-vocabulary (OOV) words. By seeing variants like "hym" in context during domain tuning, the model learns representations similar to their modern counterparts.
As shown in the results, AdaptaBERT achieved 89.8% accuracy on EME, a substantial improvement over the 85.3% achieved by standard task-tuned BERT. Notably, accuracy on OOV words jumped from 71.1% to 86.8%, nearly closing the gap with supervised in-domain training.
Social Media NER
The approach was also tested on the WNUT 2016 Named Entity Recognition (NER) task for Twitter. By incorporating 1 million unlabeled tweets during the domain tuning phase, the model's F1 score improved from 57.7 to 62.8.
Key Insights
- Handling OOV Words: The primary advantage of AdaptaBERT lies in its ability to handle words not seen in the modern training set by learning their distributional properties from unlabeled target data.
- Preserving Source Performance: Unsupervised domain tuning does not degrade performance on the original source domain (e.g., PTB). In contrast, supervised fine-tuning on small target datasets often leads to "catastrophic forgetting" of the source domain.
- Annotation Limits: The authors note that unsupervised adaptation cannot resolve differences in annotation philosophies between domains (e.g., how "to" is tagged), which remains a human-defined challenge.
Conclusion
AdaptaBERT provides a simple yet effective framework for adapting contextualized embeddings to new domains without requiring additional labeled data. While it incurs some computational cost, the gains in robustness—especially for historical and informal text—make it a valuable tool for expanding the reach of modern NLP.
Summary updated