A small Griko-Italian speech translation corpus

Marcely Zanon Boito, Antonios Anastasopoulos, Marika Lekakou, Aline Villavicencio, Laurent BesacierarXiv 2018 · arXiv

This paper introduces an extension to a tiny parallel corpus for Griko, an endangered Italo-Greek dialect spoken in Southern Italy. By providing rich annotations and baseline results for speech translation and word discovery, the authors aim to bolster the field of Computational Language Documentation (CLD) and provide a benchmark for extreme low-resource scenarios.

TL;DR

Researchers have released an enriched 20-minute speech corpus of the endangered Griko language, complete with Italian translations and word-level alignments. Their experiments demonstrate that in such extreme low-resource settings, traditional unsupervised alignment methods significantly outperform modern neural attentional models.

What Problem Does This Paper Address?

Linguists estimate that over 50% of the world’s spoken languages may vanish by the year 2100. Many of these are unwritten, making speech the only medium for preservation. While modern AI has revolutionized language processing, these "data-hungry" models typically require thousands of hours of audio.

The CLD community faces a "cold start" problem: how do we build useful computational tools for languages where we only have minutes of data? This paper addresses the lack of publicly available, high-quality datasets for these languages, specifically focusing on Griko—a language with fewer than 20,000 (mostly elderly) speakers.

Methods

The authors took an existing collection of 330 utterances (the UoI corpus) and extended it with several layers of annotation:

  • Free-form Italian translations: Created by a bilingual speaker.
  • Gold-standard alignments: Manual speech-to-transcription and speech-to-translation alignments.
  • Acoustic Unit Discovery (AUD): Using an unsupervised method to generate "pseudo-phones" (machine-discovered speech units) to simulate a scenario where no written script exists.
  • ZRC Compatibility: They formatted the data to be compatible with the Zero Resource Challenge (ZRC) metrics, allowing for standardized evaluation of word discovery.

To test the corpus, they evaluated three alignment strategies:

  1. Proportional Baseline: A naive method that segments audio based on the character length of translation words.
  2. Neural Alignment: An end-to-end sequence-to-sequence model with attention.
  3. DTW-EM: An unsupervised model that combines the IBM Model 2 alignment with Dynamic Time Warping.

Datasets

The primary dataset is the Griko-Italian speech translation corpus.

  • Size: 330 sentences (~20 minutes of speech).
  • Speakers: 9 native speakers (5 male, 4 female).
  • Content: Elicited utterances focused on specific morphosyntactic features.

As shown in Figure 1, the Griko and Italian sides of the corpus are remarkably similar in terms of average token length and sentence structure, likely due to centuries of contact between the two languages.

Table 1: Statistics of the 330 sentences in our parallel Griko-Italian corpus.

Key Insights and Ideas

  • Neural Models Struggle at Small Scales: With only 330 sentences, the neural attentional model failed to learn robust alignments, performing significantly worse than simpler baselines.
  • The Power of Language Similarity: Because Griko and Italian share similar structures, the "proportional" baseline—which assumes no reordering—is surprisingly competitive.
  • Error Propagation in Zero-Resource Pipelines: Unsupervised Word Discovery (UWD) works reasonably well on text (graphemes), but performance drops sharply when using machine-discovered pseudo-phones. This highlights the difficulty of "cascading" unsupervised systems where errors in unit discovery propagate to word discovery.

Key Results

The quantitative results highlight the gap between traditional and neural methods in this niche:

  • Speech-to-Translation Alignment: The unsupervised DTW-EM model achieved the highest F-score (53.8), while the neural model lagged behind at 27.0.
  • Word Discovery (Graphemes): The Bayesian model dpseg was the clear winner, retrieving 75.1% of correct boundaries (Recall) with a 71.6 F-score.
  • Word Discovery (Pseudo-phones): Performance was much lower across all methods. The "merged neural" approach, which averages soft-alignment matrices, proved slightly more effective here with an F-score of 30.0.
Table 3: Boundary scores for the task of unsupervised word segmentation. Results for neural segmentation are the average over 5 runs. Best results for each metric are presented in bold.

Limitations and Future Work

The most obvious limitation is the corpus size. While 20 minutes is a realistic reflection of what a field linguist might collect in a short trip, it is an extreme challenge for computational methods.

Additionally, the authors note that their speech-to-grapheme alignment is an approximation (dividing word time-windows equally among graphemes), which means subword discovery results should be interpreted with caution. Future work involves comparing these results against other low-resource corpora to see if these findings generalize across different language families.

Summary updated

Abstract

This paper presents an extension to a very low-resource parallel corpus collected in an endangered language, Griko, making it useful for computational research. The corpus consists of 330 utterances (about 20 minutes of speech) which have been transcribed and translated in Italian, with annotations for word-level speech-to-transcription and speech-to-translation alignments. The corpus also includes morphosyntactic tags and word-level glosses. Applying an automatic unit discovery method, pseudo-phones were also generated. We detail how the corpus was collected, cleaned and processed, and we illustrate its use on zero-resource tasks by presenting some baseline results for the task of speech-to-translation alignment and unsupervised word discovery. The dataset is available online, aiming to encourage replicability and diversity in computational language documentation experiments.
BibTeX
@misc{boito2018small,
  author = {Marcely Zanon Boito and Antonios Anastasopoulos and Marika Lekakou and Aline Villavicencio and Laurent Besacier},
  title = {{A small Griko-Italian speech translation corpus}},
  year = {2018},
  eprint = {1807.10740},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1807.10740},
}
Figure