DaCy: A Unified Framework for Danish NLP

Kenneth Enevoldsen, Lasse Hansen, Kristoffer Nielbo

TL;DR

DaCy is an open-source, spaCy-based framework that provides a unified, state-of-the-art pipeline for Danish Natural Language Processing (NLP). Beyond setting new performance benchmarks in core tasks, the paper introduces a data-augmentation evaluation framework that exposes how standard NLP models degrade when faced with real-world noise, spelling variations, and demographic biases.

What Problem Does This Paper Address?

While English NLP enjoys a vast ecosystem of unified tools, smaller languages like Danish often suffer from fragmentation. Prior to DaCy, Danish NLP tools were scattered across disjoint projects with conflicting dependencies and inconsistent APIs, severely hindering practical workflows.

Furthermore, existing models were evaluated solely on pristine "gold-standard" datasets. For Danish, the primary benchmark is DaNE, which contains text collected between 1983 and 1992. This creates a stark generalization gap: a model that scores perfectly on 30-year-old edited text will frequently break down when deployed on modern social media filled with typos, non-standard casing, and diverse demographic names. The authors set out to build both a unified framework and a more realistic, stress-tested evaluation methodology.

Methods

DaCy is built on top of spaCy v3, leveraging its optimized, user-friendly architecture. To maximize efficiency, DaCy uses multitask learning, predicting Part-of-Speech (POS) tags, Named Entity Recognition (NER), and dependency parsing simultaneously in a single forward pass.

The framework introduces three model sizes:

  • DaCy small: Based on a Danish ELECTRA model (14M parameters).
  • DaCy medium: Based on the Danish BERT (110M parameters).
  • DaCy large: Based on the multilingual XLM-Roberta Large (550M parameters).
Table 1: Performance of models finetuned for DaCy. Highest scores are in bold and second highest is underlined. WPS indicates words pr. second.

The paper’s most novel methodological contribution is its evaluation via data augmentation. Instead of just testing on the standard DaNE test set, the authors developed a suite of "augmenters" to systematically inject real-world variations:

  1. Keystroke errors: Simulating QWERTY keyboard typos at 2%, 5%, and 15% error rates.
  2. Character variations: Substituting historical or alternative Danish spellings (e.g., replacing æ with ae, å with aa).
  3. Casing modifications: Converting text entirely to lowercase.
  4. Name substitutions: Swapping out names to test for gender (male vs. female names) and ethnicity biases (Danish vs. Muslim names).

Datasets

  • DaNE Corpus: The standard benchmark for Danish NLP, consisting of the Danish Dependency Treebank annotated with NER tags. It serves as the baseline dataset for both training and evaluation.
  • Augmented DaNE Test Sets: Newly introduced variations of the DaNE test set generated using the stochastic and deterministic augmenters described above to evaluate robustness and bias.

Key Insights and Ideas

  • The Multilingual Robustness Paradox: Surprisingly, the largest model, DaCy large (built on the multilingual XLM-Roberta), proved significantly more robust to demographic biases than monolingual Danish models. While smaller monolingual models showed a significant performance drop when identifying names of Muslim origin compared to traditional Danish names, DaCy large maintained stable performance.
  • Benchmark Over-optimism: Evaluating models only on clean datasets masks severe vulnerabilities. Simple changes—like dropping capitalization or introducing a 2% typo rate—consistently degrade performance across all frameworks.
  • Casing is a Double-Edged Sword: Models trained on uncased text (like DaCy medium) are completely immune to lowercase variations but perform worse overall on properly cased text. Conversely, cased models dominate standard benchmarks but suffer drastically on lowercase domains like social media chat.

Key Results

DaCy large established a new state-of-the-art for Danish NLP across all evaluated tasks on the un-augmented DaNE dataset:

  • POS Tagging: 98.37% accuracy.
  • NER: 84.39% Average F1 score.
  • Dependency Parsing: 88.44% Labeled Attachment Score (LAS).
Table 2: Performance of Danish NLP pipelines. Wall Time is the time taken by the model to go through the DaNE test set without augmentation. Stanza uses the spacy-stanza implementation. The speed of the DaNLP model is reported as provided by the framework, which does not utilize batch input. However, given the model size it can be expected to reach speeds comparable to DaCy medium. Empty cells indicates that the framework does not include the specific model.

When subjected to stress testing (detailed in the paper's robustness tables), DaCy large consistently outperformed competing pipelines (such as Stanza, Flair, and Polyglot) under heavy noise. For example, at a 5% keystroke error rate, DaCy large maintained an NER F1 score of 76.9%, whereas DaCy medium dropped sharply to 50.0%.

Table 3: NER performance of Danish NLP pipelines reported as average F1 scores excluding the MISC category. Best scores are marked bold and second best are underlined. * denotes that the result is significantly different from the baseline using a significance threshold of 0.05 with Bonferroni correction for multiple comparisons. Danish names is considered the baseline for the augmentation of Muslim, female, and male names. Values in parentheses denote the standard deviation. NERDA limits input size to 128 wordpieces which leads to truncation on long input sizes and high rates of keystroke errors.

Limitations and Future Work

  • Sensitivity to Casing: DaCy large and small are highly sensitive to text casing. In entirely lowercased environments, their NER performance degrades significantly, making the uncased DaCy medium a better choice for messy social media data.
  • Evaluation, Not Training: The data augmenters in this paper were used exclusively for evaluating models. The authors note that integrating these augmenters directly into the training pipeline could significantly improve the real-world robustness of future Danish NLP models.
  • A Starting Point for Low-Resource Languages: The authors emphasize that their suite of augmenters is not exhaustive but rather a framework to encourage more thorough, bias-aware validation practices in medium- and low-resource languages.

Summary updated

Abstract

Danish natural language processing (NLP) has in recent years obtained considerable improvements with the addition of multiple new datasets and models. However, at present, there is no coherent framework for applying state-of-the-art models for Danish. We present DaCy: a unified framework for Danish NLP built on SpaCy. DaCy uses efficient multitask models which obtain state-of-the-art performance on named entity recognition, part-of-speech tagging, and dependency parsing. DaCy contains tools for easy integration of existing models such as for polarity, emotion, or subjectivity detection. In addition, we conduct a series of tests for biases and robustness of Danish NLP pipelines through augmentation of the test set of DaNE. DaCy large compares favorably and is especially robust to long input lengths and spelling variations and errors. All models except DaCy large display significant biases related to ethnicity while only Polyglot shows a significant gender bias. We argue that for languages with limited benchmark sets, data augmentation can be particularly useful for obtaining more realistic and fine-grained performance estimates. We provide a series of augmenters as a first step towards a more thorough evaluation of language models for low and medium resource languages and encourage further development.
BibTeX
@misc{enevoldsen2021dacy,
  author = {Kenneth Enevoldsen and Lasse Hansen and Kristoffer Nielbo},
  title = {{DaCy: A Unified Framework for Danish NLP}},
  year = {2021},
  eprint = {2107.05295},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2107.05295},
}
Figure