Synthesising a Corpus of Gaelic Traditional Narrative with Cross-Lingual Text Expansion

William Lamb, Dongge Han, Ondrej Klejch, Beatrice Alex, Peter BellWorkshops @ COLING 2025

Introduction

Modern Large Language Models (LLMs) and Automatic Speech Recognition (ASR) systems are data-hungry, often requiring billions of tokens for effective training. This requirement creates a severe bottleneck for Low-Resource Languages (LRLs) like Scottish Gaelic, which possesses only about 150 million digitized tokens. However, Gaelic has a rich oral tradition with thousands of hours of untranscribed historical audio. To build ASR systems for these archives, researchers need domain-specific language models. This paper introduces Cross-Lingual Text Expansion (XLTE), a method to synthetically generate high-quality training corpora by expanding short English summaries into full-length Gaelic narratives using GPT-4o.

Methodology

The XLTE technique couples text expansion with translation. The researchers prompted a multilingual LLM to take a short summary in a high-resource language (English) and generate a long-form traditional narrative in the target LRL (Gaelic).

The study compared two primary approaches:

  1. Baseline Generation: Using an off-the-shelf GPT-4o model to expand English summaries.
  2. Supervised Fine-Tuning (SFT): Fine-tuning GPT-4o on a small dataset of authentic English summary/Gaelic narrative pairs before generation.
The four-stage XLTE workflow: Summarisation, Supervised Fine-tuning, Generation, and Intrinsic Evaluation.

To evaluate the synthetic corpora, the authors trained 5-gram language models on the generated texts and measured their perplexity (PPL) against a gold-standard evaluation set of genuine human-transcribed Gaelic narratives from the School of Scottish Studies Archives (SSSA).

Key Results and Insights

The experimental results demonstrate that fine-tuning is highly effective, even with limited data. The model fine-tuned on just the 100 longest narratives (FT-100) outperformed the model trained on a larger set of 384 shorter narratives, suggesting that the quality and length of fine-tuning data are more critical than volume.

Performance metrics comparing fine-tuned models (FT-100, FT-384) against the baseline (4o-base) and genuine training data.

As shown in the results, the baseline GPT-4o model exhibited a high perplexity of 604.3. Supervised fine-tuning (FT-100) reduced this by 57.2% to 258.4. Furthermore, fine-tuning significantly reduced the rate of English code-switching and the generation of hallucinated "neologisms"—fake words created by the model's tokenizer when struggling with LRL structures.

Linguistic Observations

  • Emergent Grammar: Interestingly, the synthetic Gaelic exhibited correct gender agreement between pronouns and inanimate nouns, a feature currently fading in modern spoken Gaelic. This suggests the model learned these representations during pre-training, potentially through cross-lingual transfer from related languages like Irish.
  • Cultural Nuance: While linguistically coherent, the model struggled with domain-specific stylistic quirks, such as traditional Gaelic closing formulas, and failed to capture the full depth of folklore-specific vocabulary.

Conclusion

While synthetic text has not yet surpassed the linguistic richness of human transcription—genuine data still achieved the lowest perplexity (93.9)—XLTE provides a scalable and sample-efficient method for augmenting LRL datasets. Future work will focus on applying this method to open-weights models and exploring more diverse domains like news reportage to further reduce hallucinations and improve stylistic accuracy.

Summary updated

BibTeX
@inproceedings{lamb2025synthesising,
  author = {William Lamb and Dongge Han and Ondrej Klejch and Beatrice Alex and Peter Bell},
  title = {{Synthesising a Corpus of Gaelic Traditional Narrative with Cross-Lingual Text Expansion}},
  booktitle = {Workshops @ COLING},
  year = {2025},
}
Figure