The Classical Language Toolkit: An NLP Framework for Pre-Modern Languages

Kyle P. Johnson, Patrick J. Burns, John Stewart, Todd Cook, Clément Besnier, William J. B. MattinglyACL 2021

TL;DR

The Classical Language Toolkit (CLTK) is a new Python framework designed to bridge the gap in Natural Language Processing (NLP) for pre-modern languages. It offers a modular, customizable pipeline architecture that addresses the unique challenges of historical texts, such as limited corpora and linguistic variation, making advanced NLP techniques accessible to scholars of ancient and historical languages.

What Problem Does This Paper Address?

Modern NLP tools and algorithms are overwhelmingly built with the assumption of living languages, which have active communities of speakers, continuous text generation, and often extensive digital resources. This leaves a significant void for pre-modern languages (ancient, extinct, or historic languages whose speakers died long ago or whose written forms differ significantly from modern descendants). These languages present distinct challenges:

  • Finite and often small corpora: Unlike living languages, new texts are not being created, meaning available data can be scarce, making standard machine learning approaches difficult.
  • High variation: Pre-modern texts can exhibit non-standardized orthography, regional dialects, and significant linguistic change over centuries, complicating analysis.
  • Limited resources: Funding and development for NLP tools for these languages are typically lower due to their primarily scholarly or religious interest, rather than immediate industrial or governmental application.

Scholars of these languages often have different goals than those working with living languages, focusing on philology (historical origins of texts), historical linguistics (language change), or pedagogy (teaching difficult original texts). Existing NLP tools are not well-suited to these specific needs.

Methods

The CLTK addresses these challenges by providing a novel software architecture built around a modular processing pipeline. This pipeline is designed to be flexible and adaptable, balancing algorithmic diversity with pre-configured defaults. Key methodological aspects include:

  • Modular Pipelines: Each language has a pre-configured pipeline with default processes (e.g., tokenization, POS tagging, lemmatization). Users can easily modify, replace, or add processes, and the architecture is designed to facilitate the creation of pipelines for new languages.
  • Algorithmic Diversity: The CLTK supports multiple algorithms for common NLP tasks, acknowledging that different approaches may be more suitable depending on the language and available data. This is crucial given the often low-resource nature of pre-modern languages.
  • Standardized I/O: The framework uses a consistent API for all human languages, aiming to optimize user productivity and facilitate scholarly communication. Outputs are expressed using data structures and representations shared across languages.
  • Process Abstraction: The core of the CLTK's processing is the Process class, which serves as a base for various NLP tasks. These include NormalizeProcess, TokenizationProcess, SentenceProcess, StopsProcess, LemmatizationProcess, MorphologyProcess, DependencyProcess, NERProcess, EmbeddingsProcess, PhonologyProcess, ProsodyProcess, StemmingProcess, and WordNetProcess.
  • Data Types: Custom data types like Doc (containing processed information) and Word (holding token-level details) are used to structure and manage NLP outputs. The MorphosyntacticFeatureBundle class is used to standardize morphological annotations according to the Universal Dependencies project.

Datasets

The paper does not explicitly list specific datasets used for training or evaluation of the CLTK's models. However, it mentions that the CLTK aims to cover almost 20 languages and that its models are hosted in Git repositories by contributors. It also notes that the CLTK relies on upstream projects like Stanza for many of its pre-trained models. The paper highlights the general scarcity of large, annotated datasets for pre-modern languages, contrasting them with the massive datasets used for training models like BERT.

Key Insights and Ideas

  • NLP for the Past is Different: The core insight is that pre-modern languages have unique characteristics (finite corpora, variation, limited resources) that necessitate a specialized NLP approach, distinct from that used for living languages.
  • Modularity is Key: A flexible, modular pipeline architecture is essential for adapting NLP tools to the diverse needs and varying data availability of historical languages.
  • Bridging Scholarly and Computational Needs: The CLTK aims to serve both computational linguists and traditional humanities scholars by providing tools that can handle philological and linguistic analyses.
  • Standardization for Collaboration: By adopting standardized data types and annotation schemas (like Universal Dependencies), the CLTK facilitates better collaboration and reproducibility among researchers.
  • The "Pre-Modern" Spectrum: The CLTK defines "pre-modern" broadly to include ancient, extinct, and historic languages, encompassing a vast historical linguistic range from Sumerian to texts from the early 19th century.

Key Results

The paper announces the release of version 1.0 of the Classical Language Toolkit (CLTK). At this release, the CLTK provides pre-configured pipelines and models for almost 20 languages. The framework is available as a Python library and is MIT-licensed. The paper demonstrates the functionality of the CLTK through code examples showing tokenization, POS tagging, and feature extraction for Latin. While quantitative results on model accuracy are not presented, the architecture itself is presented as the primary contribution, enabling researchers to apply NLP to a wider range of historical texts.

Limitations and Future Work

The authors acknowledge several limitations and outline future directions:

  • Lack of Formal Evaluations: A significant limitation is the absence of formal evaluations for the CLTK's models' accuracies. Most current processes wrap models from upstream projects, and while those projects report accuracies on their training sets, external benchmarks for pre-modern languages are largely non-existent.
  • Duplicative Treebanks: There are multiple, often incompatible, treebanks for some languages (e.g., Latin, Ancient Greek), which need normalization.
  • Internet Infrastructure: The need for better internet infrastructure for training and hosting models is identified.

Future work will focus on:

  • Creating Evaluation Benchmarks: Developing benchmarks for each NLP task and language.
  • Developing a Training Pipeline: Creating a TrainingPipeline similar to the inference pipeline to standardize model training.
  • Normalizing Treebanks: Addressing the issue of duplicative treebanks.
  • Experimentation with Transfer Learning: Exploring techniques like Multilingual BERT to leverage information across related pre-modern languages and texts.

Summary updated

BibTeX
@inproceedings{johnson2021classical,
  author = {Kyle P. Johnson and Patrick J. Burns and John Stewart and Todd Cook and Clément Besnier and William J. B. Mattingly},
  title = {{The Classical Language Toolkit: An NLP Framework for Pre-Modern Languages}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
}
Figure