Modern Natural Language Processing (NLP) operates under a foundational assumption: the availability of massive, clean, and highly standardized text corpora. Large language models thrive on billions of tokens scraped from the contemporary web, where spelling rules are largely uniform and semantic definitions are relatively stable.
When we turn these tools toward historical texts, this assumption collapses. Historical documents present a severe data scarcity problem—not necessarily because the past left behind no words, but because it left behind very few digitized, standardized, and annotated words. As Kyle P. Johnson et al. point out in The Classical Language Toolkit: An NLP Framework for Pre-Modern Languages, modern NLP infrastructure is heavily optimized for living languages with active digital communities, leaving pre-modern and transitional linguistic eras severely underserved.
The Mechanics of Historical Data Scarcity
To understand why standard NLP fails on historical text, we must look at how language changes over time. Contemporary models rely on fixed vocabularies or subword tokenizers (like Byte-Pair Encoding). When confronted with historical text, these tokenizers fracture words due to two primary phenomena: orthographic variation and morphological drift.
Before the standardization of printing presses and dictionaries, spelling was highly fluid. A single word could be spelled dozens of different ways depending on the region, century, or even the scribe's personal preference. As noted by Teodora Vukovic et al. in Corpora and Processing Tools for Non-standard Contemporary and Diachronic Balkan Slavic, high orthographic and morphological variation causes standard sequence-labeling tools (like part-of-speech taggers) to fail because the model treats every spelling variant as an entirely new, unseen token.
Furthermore, the lack of raw, unstructured text prevents models from learning these variations implicitly. Jakub Pokrywka et al. observe in Challenging America: Modeling language in longer time scales that many historical archives only provide aggregated statistics rather than the raw text required to train modern deep neural architectures. When raw text is available, it often lacks the specialized annotations required for downstream tasks like Named Entity Recognition (NER) or syntactic parsing, as highlighted by Vera Provatorova et al. in Too Young to NER: Improving Entity Recognition on Dutch Historical Documents.
To overcome this data poverty, researchers have converged on four primary strategies: text normalization, continued pre-training, hybrid architectural constraints, and human-in-the-loop data bootstrapping.
Strategy 1: Text Normalization and Script Compression
If a model cannot handle the vocabulary explosion caused by historical spelling variations, the most direct solution is to map the historical text back into a standardized form before feeding it to the model.
In A Multilingual Evaluation of Three Spelling Normalisation Methods for Historical Text, Eva Pettersson et al. demonstrate that treating spelling normalization as a character-based statistical machine translation (SMT) problem significantly boosts the performance of downstream NLP tools. By learning character-level mapping rules (e.g., how specific vowels or consonants shift over centuries), the SMT system acts as a translation layer between the past and the present, effectively reducing the historical text's vocabulary to a modern standard the model already understands.
A variation of this intuition is script compression. In Tafsir Dataset: A Novel Multi-Task Benchmark for Named Entity Recognition and Topic Modeling in Classical Arabic Literature, Sajawel Ahmed et al. tackle the complexity of Classical Arabic by stripping modern diacritics and orthographic additions, reducing the text to its ancient "skeleton" form (Rasm). This compression drastically reduces vocabulary size and memory usage, allowing models to achieve high performance despite the limited availability of annotated training data.
Strategy 2: Cross-Era Transfer and Continued Pre-Training
Rather than forcing historical text into a modern mold, another approach adapts modern language models to historical contexts via continued pre-training.
Instead of training a model from scratch—which requires prohibitive amounts of data—researchers take a model already proficient in a modern language and expose it to an unannotated corpus of historical variants. Ali Al-Laith et al. apply this in Development and Evaluation of Pre-trained Language Models for Historical Danish and Norwegian Literary Texts, showing that giving a modern model a "history lesson" via continued pre-training yields substantial performance gains in downstream semantic tasks like word sense disambiguation.
However, when massive historical corpora are available, pre-training a model entirely from scratch on historical data allows it to build a native understanding of long-term language evolution. In Challenging America: Modeling language in longer time scales, the authors demonstrate that a RoBERTa model trained from scratch on centuries of digitized newspapers significantly outperforms modern counterparts at temporal tasks, such as dating documents and navigating archaic semantic shifts.
Strategy 3: Hybrid Architectures and Synthetic Augmentation
When a historical language variant is so scarce that even unannotated text is hard to find, pure deep learning models struggle to generalize. In these extreme low-resource settings, researchers inject domain knowledge through hybrid architectures or generate synthetic data.
For instance, in Lemmatisation of Medieval Greek: Against the Limits of Transformer’s Capabilities?, Colin Swaelens et al. found that state-of-the-art transformers experienced a sharp drop in performance when lemmatizing unedited Byzantine Greek. To counteract this, they implemented a hybrid system that pairs a fine-tuned transformer with a closed-class rule-based dictionary, proving that explicit linguistic constraints can compensate for a lack of training data.
Similarly, Benoît Sagot demonstrates in Développement d'un lexique morphologique et syntaxique de l'ancien français that constructing structured morphological and syntactic lexicons (such as OFrLex for Old French) provides the foundational lexical data necessary to make part-of-speech taggers viable in data-scarce environments.
When no corpus exists at all, researchers can generate one synthetically. Iglika Nikolova-Stoupak et al. address this in Compilation of a Synthetic Judeo-French Corpus by designing a pipeline that algorithmically transforms Old French texts into Judeo-French, creating a synthetic corpus to train machine translation models where no natural digital data exists.
Strategy 4: Bootstrapping via Human-in-the-Loop Pseudo-Annotation
Finally, to resolve the lack of gold-standard evaluation and training benchmarks, researchers use modern models to bootstrap historical datasets through iterative, human-verified annotation pipelines.
In Dependency Annotation of Ottoman Turkish with Multilingual BERT, Şaziye Betül Özateş et al. outline a framework where a multilingual BERT model generates initial "pseudo-annotations" for a historical language. Human experts then correct these noisy annotations, and the corrected data is fed back into the model to improve its next predictions. This iterative loop drastically reduces the manual labor required to build high-quality treebanks from scratch, offering a scalable blueprint for unlocking highly complex, low-resource historical archives.
Key Foundational Work
PCFG Parsing for Restricted Classical Chinese Texts (2002) — contributed 1 concept elements to this direction.