Rosetta Balcanica: Deriving a “Gold Standard” Neural Machine Translation (NMT) Parallel Dataset from High-Fidelity Resources for Western Balkan Languages

Edmon Begoli, Maria Mahbub, Sudarshan SrinivasanACL 2024

TL;DR

This paper introduces Rosetta Balcanica, a new parallel dataset for Western Balkan languages, aiming to create a "gold standard" for Neural Machine Translation (NMT) research in low-resource regions. By meticulously curating officially translated documents, they provide a high-quality resource that addresses the critical lack of robust training data for these languages.

What Problem Does This Paper Address?

Developing effective Neural Machine Translation (NMT) models for low-resource languages is a significant challenge due to the scarcity of high-quality parallel datasets. Many existing resources are either not meticulously translated, lack proper alignment, or are not officially sanctioned, hindering their reliability for training robust NMT systems. This paper tackles this problem by focusing on Western Balkan languages, a region often affected by crises, where international organizations produce official, high-fidelity translated documents. These documents, by their nature, are expected to be accurately translated and aligned, making them ideal candidates for a "gold standard" dataset.

Methods

The Rosetta Balcanica project employs a rigorous approach to dataset creation. The core methodology involves:

  • Resource Identification and Curation: The authors prioritize officially translated, high-fidelity documents from organizations like the Organization for Security and Cooperation in Europe (OSCE). These sources are chosen for their guaranteed professional translation and official backing.
  • Data Extraction and Preprocessing: Identified documents (primarily in PDF format) are converted into plain text. Minimal formatting is applied to remove non-linguistic elements like page numbers.
  • Alignment: Crucially, the parallel texts are aligned at the paragraph and sentence levels. This ensures that corresponding segments in different languages are accurately matched.
  • Dataset Structuring: The data is organized into a hierarchical folder structure, with language combination directories (e.g., english-macedonian-shqip) containing document-specific folders. Each document folder holds individual language files.
  • Publication and Accessibility: The curated datasets are made available through a GitHub repository and, importantly, uploaded to the Hugging Face Hub. This ensures broad accessibility for NLP researchers and practitioners.

While initial attempts at full automation for retrieval and preprocessing were found to be complex and error-prone, the current approach relies on manual collection and preprocessing for initial releases, with plans for robust automation in future stages.

Datasets

The primary dataset introduced is Rosetta Balcanica.

  • Source: The initial release focuses on parallel documents from the Organization for Security and Cooperation in Europe (OSCE), specifically for Albanian (Shqip) and Macedonian. The authors also mention plans to incorporate documents from the Hague Tribunal.
  • Characteristics: This dataset is characterized by its "gold standard" quality, derived from officially translated documents. It aims for high-fidelity translations and meticulous sentence-level alignment.
  • Relation to Existing Datasets: Unlike general web-crawled datasets (like Common Crawl or Internet Archive) which may suffer from translation quality and alignment issues, Rosetta Balcanica prioritizes official, curated sources. It aims to complement existing resources by providing a more reliable benchmark for low-resource language NMT.

Key Insights and Ideas

  • "Gold Standard" from Official Sources: The paper highlights that officially sanctioned documents from international organizations are a rich, yet underutilized, source for high-quality parallel data in low-resource languages. These documents inherently possess a level of accuracy and fidelity that is hard to achieve with other methods.
  • The Importance of Curation: The authors emphasize that meticulous curation, including accurate alignment and quality assurance, is paramount for creating truly useful NMT datasets, especially for languages with limited resources.
  • Hugging Face Hub as a Disseminator: Leveraging platforms like the Hugging Face Hub is crucial for making these specialized datasets accessible to the broader NLP community, accelerating research and development.
  • Trade-offs in Dataset Development: The paper acknowledges the trade-off between broad topic diversity and the high-quality, narrowly focused dataset they are building. This strategic choice prioritizes reliability over breadth in the initial stages.

Key Results

The paper presents the first release of the Rosetta Balcanica dataset, focusing on English, Macedonian, and Shqip.

  • Corpus Statistics: The dataset includes 8,567 sentences and 137,620 tokens for English, 148,459 for Macedonian, and 168,202 for Shqip.
  • Topic Representation: The corpus is heavily biased towards topics relevant to the OSCE's mission, such as human rights, elections, conflicts, and socio-cultural development. While this limits topic diversity, it ensures high relevance and quality for specific domains.
  • Validation: Spot validation of 1% of randomly selected samples using Google Translate confirmed the accuracy of the translations.

Limitations and Future Work

Limitations:

  • Script Diversity: A significant limitation is that Macedonian and Serbian entries are written in Cyrillic script. This requires an additional transliteration step for comparison with Latin-script languages like Croatian and Bosnian, which the authors have not yet fully implemented.
  • Topic Diversity: The initial dataset is narrowly focused on OSCE-related topics, which limits its applicability to broader domains.
  • Automation Challenges: While automation is a goal, the current manual preprocessing is time-consuming and less scalable.

Future Work:

  • Inclusion of More Languages: The project plans to expand to include other Western Balkan languages and eventually general multi-lingual resources from official media sources to diversify topics.
  • Hague Tribunal Documents: Incorporating documents from the Hague Tribunal is a planned release.
  • Full Automation: The authors intend to develop robust automation for dataset retrieval and preprocessing to improve scalability and reusability.
  • Transliteration: Addressing the Cyrillic script issue by implementing transliteration for relevant languages is a key future step.

Summary updated

BibTeX
@inproceedings{begoli2024rosetta,
  author = {Edmon Begoli and Maria Mahbub and Sudarshan Srinivasan},
  title = {{Rosetta Balcanica: Deriving a “Gold Standard” Neural Machine Translation (NMT) Parallel Dataset from High-Fidelity Resources for Western Balkan Languages}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
}
Figure