Simple Natural Language Processing Tools for Danish

Leon DerczynskiarXiv 2019 · arXiv

Introduction

In the landscape of Natural Language Processing (NLP), major languages such as English and Chinese often dominate research and development resources. Danish, spoken by approximately six million people, frequently falls into the "low-resource" category within a computational context. While Danish NLP research exists, a significant barrier has remained: the lack of accessible, easy-to-run utilities for the broader community. This technical note by Leon Derczynski introduces a suite of open-access tools—daner and dapipe—designed to provide a robust foundation for Danish NLP.

Addressing the "Artificial Lacuna"

The author identifies what is termed an "artificial lacuna"—a gap where research exists but usable tooling does not. By wrapping established frameworks and training them on Danish-specific datasets, these tools aim to lower the barrier to entry for researchers and developers. Maintained at ITU Copenhagen, these utilities provide essential capabilities like Named Entity Recognition (NER), part-of-speech tagging, and dependency parsing, bridging a significant gap in accessible resources for the Danish language community.

The Tools: daner and dapipe

daner: Named Entity Recognition

daner is built upon the Stanford CoreNLP NER component. It utilizes a machine-learning approach that analyzes contextual features, including n-grams (sequences of up to five words) and "word shapes" (e.g., identifying capitalized words as potential names).

A notable innovation in daner is the integration of Brown clusters. These clusters consist of words with similar semantic or syntactic behaviors, induced from a massive 134-million-token Danish corpus. This allows the model to generalize effectively; for instance, if the model recognizes "Aarhus" and "Copenhagen" as belonging to the same cluster, it can identify "Aarhus" as a location even if it was absent from the primary training data. daner specifically targets three main entity classes: Persons (PER), Locations (LOC), and Organizations (ORG), using the standard BIO (Beginning, Inside, Outside) tagging format.

dapipe: Structural Linguistic Analysis

dapipe serves as a wrapper for UDPipe, a standard pipeline for structural linguistic analysis. It performs three critical tasks:

  1. Tokenization: Segmenting text into sentences and individual words.
  2. Part-of-Speech (PoS) Tagging: Assigning grammatical categories (e.g., nouns, verbs, adjectives).
  3. Dependency Parsing: Establishing syntactic relationships between words to construct a "tree" representing sentence structure.

By adopting the Universal Dependencies schema, dapipe ensures that Danish NLP outputs are compatible with international research standards.

Data and Training

The efficacy of these tools relies on a diverse set of Danish datasets:

  • Copenhagen Dependency Treebank (CDT): Includes approximately 20,000 tokens of 1990s newswire.
  • Modern Newswire: 10,000 tokens of post-2015 newswire were added to ensure relevance to contemporary language.
  • Non-Newswire Danish: 3,000 tokens to provide variety beyond journalistic styles.
  • Large-Scale Crawls: For the Brown clusters, 134 million tokens were extracted from Danish Wikipedia and the Common Crawl, filtered via FastText language identification.

Insights and Practical Utility

The primary contribution of this work is the democratization of NLP. Rather than focusing solely on marginal "State of the Art" (SOTA) improvements, the author emphasizes engineering, distribution, and usability. By providing "slashed-tag" formats and straightforward scripts, the tools are accessible to non-experts.

However, the author maintains transparency regarding the tools' limitations. While robust for general use, they are heavily weighted toward newswire data. For "noisy" or colloquial text, such as social media content, the author suggests that specialized tools like structbilty may be more appropriate.

Conclusion

The release of daner and dapipe provides the Danish language community with essential, freely available baselines. While these tools represent "classical" machine learning approaches, they serve as a necessary stepping stone toward more complex Transformer-based models. These resources are hosted and maintained at ITU Copenhagen, fulfilling the goal of providing persistent, high-quality linguistic tools for the Danish language.

Summary updated

Abstract

This technical note describes a set of baseline tools for automatic processing of Danish text. The tools are machine-learning based, using natural language processing models trained over previously annotated documents. They are maintained at ITU Copenhagen and will always be freely available.
BibTeX
@misc{derczynski2019simple,
  author = {Leon Derczynski},
  title = {{Simple Natural Language Processing Tools for Danish}},
  year = {2019},
  eprint = {1906.11608},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1906.11608},
}
Figure