A Preliminary Exploration of Phrase-Based SMT and Multi-BPE Segmentations through Concatenated Tokenised Corpora for Low-Resource Indian Languages

Saumitra Yadav, Manish Shrivastava2025 · DOI

TL;DR

In the age of massive neural networks, researchers from IIIT Hyderabad took a "back-to-basics" approach for the WMT 2025 Shared Task. They explored whether traditional Phrase-Based Statistical Machine Translation (SMT), enhanced by a novel "Multi-BPE" data augmentation strategy, could outperform data-hungry neural models for extremely low-resource Indian languages like Nyishi, Khasi, and Assamese.

What Problem Does This Paper Address?

Neural Machine Translation (NMT) has revolutionized the field, but it has a major Achilles' heel: it requires mountains of parallel data to function effectively. For many Indian languages, such data is nearly non-existent. When data is scarce, models suffer from "out-of-vocabulary" (OOV) issues—where the system simply doesn't know the words it's seeing—and poor generalization.

The authors address this by revisiting Statistical Machine Translation (SMT). Unlike neural models, SMT is often more resilient to small datasets because it relies on explicit statistical alignments of phrases. The challenge they tackle is how to further squeeze performance out of these small datasets, specifically by improving subword segmentation to better handle the complex grammar and word structures (morphology) of Indic languages.

Methods

The researchers combined a classic SMT pipeline with a clever data-processing "twist":

  1. The Engine: They used the Moses toolkit, the industry standard for phrase-based SMT, paired with KenLM for language modeling.
  2. Multi-BPE Augmentation: This is the core innovation. Usually, researchers pick a single "merge" value for Byte Pair Encoding (BPE) to break words into sub-units. Instead, these authors applied four different BPE merge operations (500, 1000, 2000, and 3000).
  3. Concatenation: They concatenated the outputs of all four BPE settings into a single, much larger training set. This effectively "augmented" the data by showing the model the same sentences segmented in multiple ways, theoretically improving its vocabulary coverage and alignment robustness.
  4. Inference Strategies: During translation, they tested the system using specific BPE segmentations and a "Combined Hypothesis" method, which selected the translation with the highest probability across different configurations.

Datasets

The study utilized parallel corpora provided by the WMT 2025 Shared Task organizers. The primary focus was on three language pairs:

  • English–Khasi: ~26,000 original sentences (expanded to ~91,000 via Multi-BPE).
  • English–Mizo: ~50,000 original sentences (expanded to ~187,000).
  • English–Assamese: ~54,000 original sentences (expanded to ~209,000).
  • English–Nyishi: Also included in evaluation, though it represented the most extreme data scarcity.

Key Insights and Ideas

  • Subword Diversity as Regularization: By using multiple BPE levels, the authors essentially forced the SMT model to learn alignments at different granularities—from nearly character-level to full-word level. This acts as a form of data augmentation that helps the model "guess" the meaning of rare word forms.
  • SMT’s Floor: The paper reinforces the idea that SMT provides a reliable "floor" or baseline for low-resource settings. While it doesn't have the "magic" of neural models, it doesn't collapse as easily when data is measured in thousands rather than millions of sentences.
  • The Limits of Tradition: A key takeaway is that while Multi-BPE helps, it cannot fully compensate for SMT's inherent inability to capture long-range linguistic dependencies—a feat that modern Transformer-based NMT models excel at.

Key Results

The quantitative results were modest, highlighting the extreme difficulty of the task:

  • BLEU Scores: The highest scores were achieved in the English-to-Khasi direction (4.2570) and English-to-Assamese (3.0287).
  • Indic-to-English: Performance was significantly lower, with Assamese-to-English struggling at a BLEU score of just 0.3340.
  • Comparison: The authors noted that their SMT-based systems ranked in the "lower tier" compared to top-performing NMT systems in the shared task, which likely utilized large-scale pre-trained models or more sophisticated external data.

Limitations and Future Work

The authors are candid about the limitations of this exploratory work:

  1. Context Blindness: SMT's phrase-based nature means it struggles with the nuanced, long-distance patterns found in natural language.
  2. Data Scarcity: Even with Multi-BPE augmentation, the "extreme scarcity" of data (especially for Nyishi) proved too high a hurdle for the system to achieve competitive fluency.
  3. Noise: The provided corpora contained noisy and domain-specific terms that likely degraded the statistical alignment quality.

Future Directions: The team plans to move toward hybrid SMT-NMT systems, explore back-translation (using synthetic data to boost training), and investigate fine-tuning large multilingual models that can transfer knowledge from high-resource languages to these underserved Indic languages.

Summary updated

Abstract

This paper describes our methodology and findings in building Machine Translation (MT) systems for submission to the WMT 2025 Shared Task on Low-Resource Indic Language Translation.Our primary aim was to evaluate the effectiveness of a phrase-based Statistical Machine Translation (SMT) system combined with a less common subword segmentation strategy for languages with very limited parallel data.We applied multiple Byte Pair Encoding (BPE) merge operations to the parallel corpora and concatenated the outputs to improve vocabulary coverage.We built systems for the English-Nyishi, English-Khasi, and English-Assamese language pairs.Although the approach showed potential as a data augmentation method, its performance in BLEU scores was not competitive with other shared task systems.This paper outlines our system architecture, data processing pipeline, and evaluation results, and provides an analysis of the challenges, positioning our work as an exploratory benchmark for future research in this area.
BibTeX
@misc{yadav2025preliminary,
  author = {Saumitra Yadav and Manish Shrivastava},
  title = {{A Preliminary Exploration of Phrase-Based SMT and Multi-BPE Segmentations through Concatenated Tokenised Corpora for Low-Resource Indian Languages}},
  year = {2025},
  doi = {10.18653/v1/2025.wmt-1.103},
}
Figure