TL;DR
DaCy is an open-source, spaCy-based framework that provides a unified, state-of-the-art pipeline for Danish Natural Language Processing (NLP). Beyond setting new performance benchmarks in core tasks, the paper introduces a data-augmentation evaluation framework that exposes how standard NLP models degrade when faced with real-world noise, spelling variations, and demographic biases.
What Problem Does This Paper Address?
While English NLP enjoys a vast ecosystem of unified tools, smaller languages like Danish often suffer from fragmentation. Prior to DaCy, Danish NLP tools were scattered across disjoint projects with conflicting dependencies and inconsistent APIs, severely hindering practical workflows.
Furthermore, existing models were evaluated solely on pristine "gold-standard" datasets. For Danish, the primary benchmark is DaNE, which contains text collected between 1983 and 1992. This creates a stark generalization gap: a model that scores perfectly on 30-year-old edited text will frequently break down when deployed on modern social media filled with typos, non-standard casing, and diverse demographic names. The authors set out to build both a unified framework and a more realistic, stress-tested evaluation methodology.
Methods
DaCy is built on top of spaCy v3, leveraging its optimized, user-friendly architecture. To maximize efficiency, DaCy uses multitask learning, predicting Part-of-Speech (POS) tags, Named Entity Recognition (NER), and dependency parsing simultaneously in a single forward pass.
The framework introduces three model sizes:
- DaCy small: Based on a Danish ELECTRA model (14M parameters).
- DaCy medium: Based on the Danish BERT (110M parameters).
- DaCy large: Based on the multilingual XLM-Roberta Large (550M parameters).
The paper’s most novel methodological contribution is its evaluation via data augmentation. Instead of just testing on the standard DaNE test set, the authors developed a suite of "augmenters" to systematically inject real-world variations:
- Keystroke errors: Simulating QWERTY keyboard typos at 2%, 5%, and 15% error rates.
- Character variations: Substituting historical or alternative Danish spellings (e.g., replacing æ with ae, å with aa).
- Casing modifications: Converting text entirely to lowercase.
- Name substitutions: Swapping out names to test for gender (male vs. female names) and ethnicity biases (Danish vs. Muslim names).
Datasets
- DaNE Corpus: The standard benchmark for Danish NLP, consisting of the Danish Dependency Treebank annotated with NER tags. It serves as the baseline dataset for both training and evaluation.
- Augmented DaNE Test Sets: Newly introduced variations of the DaNE test set generated using the stochastic and deterministic augmenters described above to evaluate robustness and bias.
Key Insights and Ideas
- The Multilingual Robustness Paradox: Surprisingly, the largest model, DaCy large (built on the multilingual XLM-Roberta), proved significantly more robust to demographic biases than monolingual Danish models. While smaller monolingual models showed a significant performance drop when identifying names of Muslim origin compared to traditional Danish names, DaCy large maintained stable performance.
- Benchmark Over-optimism: Evaluating models only on clean datasets masks severe vulnerabilities. Simple changes—like dropping capitalization or introducing a 2% typo rate—consistently degrade performance across all frameworks.
- Casing is a Double-Edged Sword: Models trained on uncased text (like DaCy medium) are completely immune to lowercase variations but perform worse overall on properly cased text. Conversely, cased models dominate standard benchmarks but suffer drastically on lowercase domains like social media chat.
Key Results
DaCy large established a new state-of-the-art for Danish NLP across all evaluated tasks on the un-augmented DaNE dataset:
- POS Tagging: 98.37% accuracy.
- NER: 84.39% Average F1 score.
- Dependency Parsing: 88.44% Labeled Attachment Score (LAS).
When subjected to stress testing (detailed in the paper's robustness tables), DaCy large consistently outperformed competing pipelines (such as Stanza, Flair, and Polyglot) under heavy noise. For example, at a 5% keystroke error rate, DaCy large maintained an NER F1 score of 76.9%, whereas DaCy medium dropped sharply to 50.0%.
Limitations and Future Work
- Sensitivity to Casing: DaCy large and small are highly sensitive to text casing. In entirely lowercased environments, their NER performance degrades significantly, making the uncased DaCy medium a better choice for messy social media data.
- Evaluation, Not Training: The data augmenters in this paper were used exclusively for evaluating models. The authors note that integrating these augmenters directly into the training pipeline could significantly improve the real-world robustness of future Danish NLP models.
- A Starting Point for Low-Resource Languages: The authors emphasize that their suite of augmenters is not exhaustive but rather a framework to encourage more thorough, bias-aware validation practices in medium- and low-resource languages.
Summary updated