Overcoming Data Scarcity in Historical Linguistics

30 concepts29 papers2 proposals

Modern natural language processing thrives on abundance. Large language models and deep neural networks typically require millions or billions of tokens to learn stable representations of grammar, syntax, and semantics. However, when we turn our computational tools toward historical, classical, or extinct languages, we hit a hard physical wall: the finite horizon of surviving text.

Whether dealing with medieval manuscripts, clay tablets, or early printings, historical corpora are frequently limited to a few thousand tokens or examples, as noted by Zampieri et al. and Bollmann et al.. This extreme data sparsity causes standard deep learning architectures to overfit to noise, destabilizes word embeddings (Kutuzov et al.), and prevents models from generalizing beyond the specific archaic forms seen during training (Flachs et al.).

To bridge this gap, researchers in computational historical linguistics have developed a suite of techniques designed to maximize the utility of scarce data. By leveraging transfer learning, injecting structural and linguistic priors, and adapting neural architectures for data efficiency, these models can uncover patterns in languages that have not been spoken for millennia.

Borrowing Strength via Transfer Learning

If a model cannot learn a language from scratch due to a lack of data, it can instead "borrow" statistical strength from modern descendants or multilingual relatives. Languages evolve, but they retain deep structural, syntactic, and semantic DNA over centuries. Transfer learning exploits this continuity.

Instead of training models from a blank slate, researchers frequently adapt existing architectures. For instance, Grobol et al. demonstrate that "post-training" contemporary French and multilingual BERT models on a small corpus of medieval text yields highly effective part-of-speech tagging and dependency parsing for Old French. Similarly, Yamshchikov et al. fine-tune BERT models to capture the nuances of Ancient Greek, allowing them to successfully address long-standing authorship mysteries in classical literature.

[Massive Modern/Multilingual Corpus] ---> Pre-trains Large Model (Learns Syntax/Semantics)
                                                    |
                                                    v
[Scarce Historical Corpus] -------------> Fine-tunes Model (Adapts to Archaic Forms)

This transfer of strength operates at multiple scales. On a dialectal level, Bollmann et al. use multi-task learning to share knowledge across related historical dialects, improving spelling normalization even when a specific target dialect has minimal data. At the largest scale, Akavarapu et al. show that massive state-of-the-art large language models (LLMs) exhibit surprising zero-shot generalization capabilities on classical languages like Sanskrit and Latin. Because these massive models capture broad cross-lingual patterns during pre-training, they can outperform smaller, specialized models that have been fine-tuned on limited historical datasets.

Grounding Models with Structural and Linguistic Priors

Purely data-driven models assume nothing about the world and attempt to learn every rule from statistical co-occurrences. In low-resource settings, this assumption fails because the data does not contain enough examples to rule out incorrect hypotheses. To counteract this, researchers inject inductive biases—linguistic or geometric constraints—directly into the learning process.

When modeling lexical borrowing across historical languages, Tsvetkov et al. incorporate prior knowledge of universal linguistic structures. By forcing the model to operate within these universal constraints, it can generalize from minimal evidence rather than requiring massive parallel corpora to discover word-to-word mappings.

This principle also explains why neural models excel at tasks like cognate prediction and protoform reconstruction. As Fourrier et al. and Sagot et al. demonstrate, multilingual cognate prediction models succeed because they latently capture historical sound changes and phonetic structures. The models do not need to see millions of examples because the underlying mathematical space is constrained by the physical realities of human speech and systematic phonetic drift.

Without Priors:  [Scarce Data] ---> Infinite Hypotheses ---> Overfitting / Noise
With Priors:     [Scarce Data] + [Linguistic Constraints] ---> Narrow Search Space ---> Correct Generalization

Structural priors can also be geometric. Ancient scripts are often preserved on damaged physical mediums like clay tablets. To classify rare cuneiform signs where labeled data is virtually non-existent, Mikulinsky et al. introduce ProtoSnap, an unsupervised tool that "snaps" idealized font skeletons onto messy, real-world photographs. By anchoring the model to the known internal geometry of the signs, the system can generate high-quality synthetic training data, bypassing the data scarcity bottleneck entirely.

Designing Data-Efficient Architectures

When data is scarce, scaling up the number of model parameters can be counterproductive. Larger models have higher capacity, meaning they can easily memorize small datasets instead of learning the underlying rules. Consequently, historical linguistics often requires tailoring architectures to be inherently data-efficient.

In an evaluation of historical spelling normalization across multiple languages, Tang et al. found that simpler, vanilla Recurrent Neural Networks (RNNs) frequently remain competitive with or outperform complex, data-hungry Transformer models. Because historical spelling normalization typically involves short sequences and limited training pairs, the lower capacity of an RNN acts as a natural regularizer.

Where Transformers do succeed in low-resource historical tasks, it is often because the architecture is forced to learn structural relationships rather than raw token frequencies. For example, Chang et al. successfully applied Transformers to protoform reconstruction with only a few hundred cognate sets. The model succeeded because its internal attention mechanisms implicitly captured the "family tree" relationships between the descendant languages, organizing its embedding space to mirror historical phylogeny.

Another architectural strategy is representation disentanglement. When analyzing the handwriting (scribal hands) of the ancient Linear B script, Srivatsan et al. designed a fully unsupervised neural network that explicitly separates the content (the character shape) from the style (the scribe's unique handwriting) into independent vector spaces. By preventing these two factors from confounding each other, the model achieves robust generalization from only a few thousand surviving inscriptions.

Bootstrapping Resources through Distant Supervision

In many cases, historical linguists lack not only raw text but also the gold-standard annotations (such as part-of-speech tags or syntactic trees) required to train supervised models. To overcome this, researchers use distant supervision and standardized frameworks to bootstrap new resources from scratch.

[Modern Translation] --------> [Word Alignment] --------> [Historical Text]
         |                                                       |
         v                                                       v
[Modern POS Tagger] --------> [Project Labels] ---------> [Bootstrapped Historical Tagger]

Sukhareva et al. demonstrated this by building the first statistical part-of-speech tagger for Hittite, an extinct Indo-European language. Because manually annotated Hittite data was virtually non-existent, the authors used distant supervision, projecting grammatical labels from modern German translations back onto the Hittite cuneiform text via word alignments, combined with an unsupervised stemming technique.

To ensure that these hard-won, low-resource datasets can be combined and utilized by the broader scientific community, researchers rely on unified annotation frameworks. Zeldes et al. developed the first Universal Dependencies treebank for Coptic by adapting global syntactic guidelines to the specific structural realities of the ancient Egyptian language. By standardizing scarce data, these efforts ensure that ancient languages can interface directly with modern computational tools, maximizing the impact of every surviving word.

Key Foundational Work

Modeling Language Change in Historical Corpora: The Case of Portuguese (2016) — contributed 1 concept elements to this direction.
Improving historical spelling normalization with bi-directional LSTMs and multi-task learning (2016) — contributed 1 concept elements to this direction.
Constraint-Based Models of Lexical Borrowing (2015) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_data_scarcity_in_historical_linguistics_86aa3cbe,
  title = {data scarcity in historical linguistics},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/data-scarcity-in-historical-linguistics/rdr_86aa3cbeed7d426b92bf5093bcf2d424},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_data_scarcity_in_historical_linguistics_86aa3cbe_v1,
  title = {data scarcity in historical linguistics},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/data-scarcity-in-historical-linguistics/rdr_86aa3cbeed7d426b92bf5093bcf2d424/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake