The ParCoLab Parallel Corpus and Its Extension to Four Regional Languages of France

Dejan Stosic, Saša Marjanović, Delphine Bernhard, Xavier Bach, Myriam Bras, Laurent Kevers, Stella Retali-Medori, Marianne Vergez-Couret, Carole WernerCOLING 2024

TL;DR

This paper introduces ParCoLab, a parallel corpus and web platform designed to support low-resource languages. They've expanded it to include four regional languages of France (Alsatian, Corsican, Occitan, and Poitevin-Saintongeais), creating a valuable new resource for linguistic research, machine translation, and language learning.

What Problem Does This Paper Address?

Creating parallel corpora – collections of texts and their translations – is crucial for developing tools like machine translation and for linguistic analysis. However, for many of the world's languages, especially "low-resource" ones, these corpora are scarce. This problem is particularly acute for regional languages within larger countries. For instance, regional languages of France are significantly underserved compared to French itself, often lacking digital resources and tools. While some efforts exist, like the Tatoeba collection, the number of parallel sentences for these languages is often very small. Furthermore, existing web-crawled corpora can be of poor quality, and manual collection is time-consuming and expensive. The paper aims to address this gap by building and extending a robust parallel corpus and platform for these underrepresented languages.

Methods

The core of the paper is the ParCoLab platform and its corpus. The platform itself is built on an ElasticSearch engine, allowing for versatile querying of parallel texts across multiple languages. The corpus is constructed through a multi-stage process:

  • Initial Corpus Development: Started with Serbian, French, and English, later adding Spanish.
  • Extension to Regional Languages: A pilot study for Occitan was conducted, followed by the inclusion of Alsatian, Corsican, and Poitevin-Saintongeais.
  • Text Selection Criteria: Texts were chosen based on availability of translations in at least two target regional languages, copyright status (prioritizing open access), translation difficulty, potential for manual annotation, relevance to contemporary issues, and reusability in pedagogical contexts.
  • Collection and Translation: Texts were collected, distinguishing between public domain works, existing published translations (negotiating rights), and new translations commissioned for the project.
  • Alignment: Sentence alignment is performed using an in-house tool within ParCoLab, with manual checking and correction. For some texts, external tools like InterText are also used, with alignments converted to TEI XML format.

Datasets

The paper introduces the "DIVITAL parallel corpus" as an extension of the existing ParCoLab corpus. This new corpus specifically focuses on four regional languages of France: Alsatian, Corsican, Occitan, and Poitevin-Saintongeais, alongside French. The corpus contains over 20,000 tokens per regional language. The paper also references existing datasets and projects, such as the Tatoeba collection, FLORES-200, and various treebanks (French, English, Serbian) used for annotation processes.

Key Insights and Ideas

  • The "Regional Language Gap" is Significant: The paper clearly illustrates the disparity in digital resources between national languages (like French) and regional languages within France, using a 6-level taxonomy to highlight this.
  • Multi-faceted Approach to Corpus Building: The authors emphasize that building a useful corpus for low-resource languages requires careful consideration of text selection, copyright, translation quality, and alignment accuracy, going beyond simple web crawling.
  • ParCoLab as a Flexible Solution: The platform is designed to be adaptable, offering different configurations for hosting regional languages depending on whether a native textual database already exists.
  • The Importance of Human Curation: Despite automated alignment tools, the paper stresses the necessity of manual checking and correction to ensure the high quality and reliability of the parallel corpus.
  • Bridging the Gap for NLP and Linguistics: The ParCoLab corpus and platform serve a dual purpose: providing resources for NLP development (like machine translation) and enabling contrastive linguistic studies.

Key Results

The paper details the creation of a new parallel corpus for four regional languages of France, adding over 20,000 tokens per language. The ParCoLab platform now hosts these languages, offering a user-friendly interface for querying and accessing the data. The corpus is structured to facilitate cross-lingual comparisons and is intended to be a valuable resource for language learners, teachers, and researchers.

Limitations and Future Work

  • Limited Diversity: The collected corpus for regional languages of France currently lacks diversity in terms of author gender, language variants, genres, time periods, and source languages.
  • Copyright Restrictions: Some texts are still under copyright, limiting their shareability and availability to only within the ParCoLab platform with restricted context viewing.
  • Future Work: The authors plan to manually annotate parts of the regional language texts using the Universal Dependencies framework to provide richer annotation layers for querying and contrastive studies. They also aim to enhance the user interface with more metadata parameters and to continue expanding the corpus with more texts and languages.

Summary updated

BibTeX
@inproceedings{stosic2024parcolab,
  author = {Dejan Stosic and Saša Marjanović and Delphine Bernhard and Xavier Bach and Myriam Bras and Laurent Kevers and Stella Retali-Medori and Marianne Vergez-Couret and Carole Werner},
  title = {{The ParCoLab Parallel Corpus and Its Extension to Four Regional Languages of France}},
  booktitle = {International Conference on Computational Linguistics},
  year = {2024},
}
Figure