Efficacy of ByT5 in Multilingual Translation of Biblical Texts for Underrepresented Languages

Corinne Aars, Lauren Adams, Xiaokan Tian, Zhaoyu Wang, Colton Wismer, Jason Wu, Pablo Rivas, Korn Sooksatra, Matthew FendtarXiv 2024 · DOI · arXiv

Abstract

Researchers at Baylor University have fine-tuned the ByT5 model on a massive multilingual Bible corpus to accelerate the translation of sacred texts into underrepresented languages. By utilizing byte-level tokenization, the model effectively handles the complex morphology and unique scripts of low-resource languages, achieving a BLEU score of 0.27 and providing a viable "first-draft" tool for human translators.

Introduction

Translating sacred texts like the Bible into less commonly spoken languages is a monumental task. Traditionally, this process relies on a handful of bilingual experts and can take decades to produce a single complete version. The primary bottleneck is the "linguistic divide": most modern Natural Language Processing (NLP) models are optimized for high-resource languages (like English or Spanish) and struggle with the "long tail" of the world's 7,000+ languages.

Underrepresented languages often lack the massive datasets required to train standard subword-based models. Furthermore, sacred texts require a specific formal lexicon and structure that general-purpose translation engines (like Google Translate) often fail to capture accurately. This research seeks to bridge this gap by leveraging the universal structure of the Bible as a training foundation.

Methodology

The authors employed ByT5, a "token-free" version of Google’s T5 (Text-to-Text Transfer Transformer) model, which offers several advantages for low-resource translation.

Byte-Level Tokenization

Unlike standard models that break text into words or subwords—which requires a pre-defined vocabulary—ByT5 operates on raw UTF-8 bytes. This makes it exceptionally robust for languages with complex morphology (where words change form frequently) or character-intensive scripts (like Chinese or Japanese) that do not use clear word boundaries. By avoiding word-level tokenization, the model demonstrates enhanced generalizability; it does not get "confused" by rare words or unique character combinations, allowing it to learn the underlying patterns of sparse languages more effectively.

Fine-Tuning and Human-in-the-Loop

The model was trained using an iterative experimental protocol with a learning rate of 0.0002 and a batch size of 48. The researchers emphasize that the model is intended to complement rather than replace human translators. By providing a high-quality initial draft, the tool can significantly shorten the overall project timeline for translation teams working in the field.

Datasets: The Bible as a Universal Benchmark

The study utilized the Johns Hopkins University Bible Corpus, a massive parallel dataset containing over 4,000 versions of the Bible in more than 1,600 languages, totaling 3 million data samples. Because the Bible is the most translated text in history and follows a rigid book/chapter/verse structure, it serves as an ideal "Rosetta Stone" for training models on languages that have almost no other digital footprint. By training specifically on this corpus, the ByT5 model learns to replicate the specific tone, archaic vocabulary, and formal structure required for religious literature.

Key Results and Qualitative Analysis

The model achieved a BLEU score of 0.27. While this may seem low compared to high-resource pairs like English-to-French, it is a significant achievement for underrepresented languages where benchmarks are often non-existent. Byte-level tokenization also resulted in a smaller vocabulary size, contributing to more efficient training and lower memory requirements.

Qualitatively, the model successfully captured the semantic meaning and formal tone across a variety of languages. For instance, in Marathi—an underrepresented language in this context—the model demonstrated strong performance in replicating the target text's nuances.

Table 1: Comparison of target and translated texts for English, Spanish, Marathi, and Chinese.

The model's versatility was further demonstrated across other major languages and scripts, including German, French, Russian, and Japanese, showing a consistent ability to handle diverse linguistic structures and scripts.

Table 2: Comparison of target and translated texts for English, German, French, Russian, and Japanese.

Limitations and Future Work

The authors acknowledge that the BLEU score is a "noisy" metric for sacred texts, as it measures word overlap but doesn't fully capture the linguistic fidelity or theological accuracy essential for religious communities. Additionally, data imbalance remains a challenge; while the corpus is vast, some languages are only represented by a single book of the Bible, limiting the model's depth in those cases.

Future directions include integrating ByT5 with other NLP tools to better handle idiomatic expressions and broader context. The team also aims to expand the methodology to other significant cultural and educational texts to further support global linguistic diversity.

Summary updated

Abstract

This study presents the development and evaluation of a ByT5-based multilingual translation model tailored for translating the Bible into underrepresented languages. Utilizing the comprehensive Johns Hopkins University Bible Corpus, we trained the model to capture the intricate nuances of character-based and morphologically rich languages. Our results, measured by the BLEU score and supplemented with sample translations, suggest the model can improve accessibility to sacred texts. It effectively handles the distinctive biblical lexicon and structure, thus bridging the linguistic divide. The study also discusses the model's limitations and suggests pathways for future enhancements, focusing on expanding access to sacred literature across linguistic boundaries.
BibTeX
@misc{aars2024efficacy,
  author = {Corinne Aars and Lauren Adams and Xiaokan Tian and Zhaoyu Wang and Colton Wismer and Jason Wu and Pablo Rivas and Korn Sooksatra and Matthew Fendt},
  title = {{Efficacy of ByT5 in Multilingual Translation of Biblical Texts for Underrepresented Languages}},
  year = {2024},
  eprint = {2405.13350},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2405.13350},
}
Figure