Abstract
We apply a simple method to machine translate datasets with span-level annotation using the DeepL MT service and its ability to translate formatted documents. Using this method, we produce a Finnish version of the SQuAD2.0 question answering dataset and train QA retriever models on this new dataset. We evaluate the quality of the dataset and more generally the MT method through direct evaluation, indirect comparison to other similar datasets, a backtranslation experiment, as well as through the performance of downstream trained QA models. In all these evaluations, we find that the method of transfer is not only simple to use but produces consistently better translated data. Given its good performance on the SQuAD dataset, it is likely the method can be used to translate other similar span-annotated datasets for other tasks and languages as well.
Introduction
In extractive Question Answering (QA), models are trained to identify the exact "span"—the start and end indices—of an answer within a given text passage. While English-language research benefits from massive datasets like SQuAD 2.0, many other languages, including Finnish, remain "low-resource" in this domain.
The standard approach to bridging this gap is Machine Translation (MT). However, translating span-annotated data is notoriously difficult. Translating the passage and answer separately often leads to inconsistencies where the translated answer does not appear verbatim in the translated passage. Conversely, translating them together usually results in the loss of span markers. Previous solutions have relied on complex "translate-align-retrieve" pipelines that are prone to noise and significant data loss, sometimes discarding over 30% of the original dataset due to alignment failures.
Methodology: Markup-based Transfer
We propose a "Markup-based Transfer" method that leverages the ability of commercial MT services to preserve document formatting. Instead of complex word-alignment algorithms, we treat the MT engine as a black box that respects semantic metadata encoded as visual styles.
- Encoding: The English SQuAD dataset is converted into a formatted document (e.g., a Word document) where each answer span is assigned a unique text color.
- Translation: The document is submitted to the DeepL MT service. DeepL translates the text while preserving the color formatting in the output.
- Extraction: The translated spans are programmatically extracted based on their colors to reconstruct the dataset in Finnish.
- Handling Overlaps: In cases where answers overlap, the overlapping region is assigned its own unique color and merged back during the reconstruction phase.
This approach ensures that answers are translated in context, resulting in more natural language than isolated phrase translation.
Experimental Results
We created FinSQuAD, a Finnish version of SQuAD 2.0, which preserved 90,233 question-answer pairs—representing 97.2% of the original English dataset. This is a significant improvement over previous Finnish efforts that lost nearly 30% of the data.
Performance Comparison
We evaluated our dataset by training a FinBERT model and comparing it against models trained on other translated SQuAD versions. As shown in the results below, our method significantly outperforms previous alignment-based approaches for Finnish.
A FinBERT model trained on our dataset achieved an F1 score of 73.7, a substantial increase from the 61.9 F1 achieved by models trained on the previous Finnish SQuAD (Kylliäinen, 2022). For context, the original English BERT-base achieves an F1 of 77.6 on the English SQuAD 2.0.
Manual Validation
To assess the quality of the span transfer, we conducted a manual error analysis of 321 questions. The results indicate that the markup-based method is highly reliable.
Approximately 87.2% of the spans were perfectly transferred. The most frequent error (9%) was "over-extension," where the MT engine included an extra trailing token or punctuation mark in the colored span. Only 2.2% of spans failed to transfer entirely.
Discussion and Limitations
While highly effective, the method has some limitations. It relies on DeepL, a proprietary service, which may limit reproducibility if the service's behavior changes. Additionally, SQuAD requires answers to be single continuous strings. In Finnish, grammar sometimes naturally splits an answer across a sentence; in these cases (2.6% of the data), we forced the spans to be continuous, introducing minor noise.
Through a backtranslation experiment (Finnish to English), we estimated the "translation penalty"—the noise introduced by the MT process itself. We observed a drop of approximately 5.1 F1 points compared to the original English data, which we consider an acceptable trade-off for a dataset produced with zero human annotation at a cost of only €20.
Conclusion
The markup-based transfer method is an elegant, low-cost solution for creating high-quality span-annotated datasets in non-English languages. Given its success with SQuAD, we anticipate this method can be extended to other tasks such as Named Entity Recognition (NER) and Relation Extraction across the 33 languages currently supported by DeepL.
Summary updated