Mind the Gap: Diverse NMT Models for Resource-Constrained Environments

Ona de Gibert, Dayyán O'Brien, Dusan Varis, Jörg TiedemannNoDaLiDa/Baltic-HLT 2025 · OpenReview

Abstract

We present fast Neural Machine Translation (NMT) models for 17 diverse languages, developed using Sequence-level Knowledge Distillation. Our selected languages span multiple language families and scripts, including low-resource languages. The distilled models achieve comparable performance while being 10x faster than Transformer-base and 35x faster than Transformer-big architectures. Our experiments reveal that teacher model quality and capacity strongly influence distillation success, as well as the language script. We also explore the effectiveness of multilingual students.

Introduction

While Large Language Models (LLMs) currently dominate the AI landscape, their massive computational requirements make them impractical for on-device translation, low-latency applications, and environments with strict privacy or energy constraints. Furthermore, while LLMs excel at high-resource languages, they often struggle with low-resource pairs. This paper addresses the "efficiency gap" by focusing on traditional sequence-to-sequence (seq2seq) Transformer architectures, aiming to create models small enough for modest hardware but capable of handling 17 languages across 13 families and 7 scripts.

Methodology: Interpolated Sequence-Level Knowledge Distillation

The core technique employed is Interpolated Sequence-Level Knowledge Distillation (Seq-KD). This process involves a high-performing "teacher" model (an OPUS-MT Transformer) and a compact "student" model.

Overview of the Interpolated Sequence-Level Knowledge Distillation (Seq-KD) process.

The distillation process follows three main steps:

  1. Candidate Generation: The teacher model generates an "8-best" list of candidate translations for the training data.
  2. Interpolation: The candidate that most closely matches the original human reference (determined by the BLEU metric) is selected as the target for training the student.
  3. Architecture Design: Student models utilize a "thin and deep" design, featuring a 6-layer Transformer encoder and a lightweight 2-layer decoder using Simpler Simple Recurrent Units (SSRU). Finally, models are quantized to 8-bit integers to minimize memory footprint.

Experimental Setup

The study utilized a wide variety of open-source corpora, primarily from the Tatoeba Challenge. To bolster specific low-resource or regional languages, specialized corpora like MaCoCu (Catalan), CLUVI (Galician), and SAWA/Gourmet (Swahili) were included. Evaluation was conducted using the Flores-200 benchmark to ensure standardized results across all 17 languages.

Results and Discussion

The primary achievement of this work is the drastic reduction in model size and increase in inference speed without significant loss in translation quality.

Comparison of teacher and student model architectures, parameter counts, and inference speeds.

Performance and Efficiency

The "tiny" student models, compressed to just 17M parameters and 17MB in size, are 10x faster than Transformer-base and 35x faster than Transformer-big models. On average, students experienced a drop of only 2.9 COMET points compared to their teachers. Notably, the Catalan student model actually outperformed its teacher by 1.1 COMET points.

Key Insights

  • The Capacity Gap Paradox: Distilling from a "Big" teacher resulted in a performance drop twice as large as distilling from a "Base" teacher. This suggests that if the capacity gap is too wide, the student cannot effectively mimic the teacher's complex internal representations.
  • Script Influence: Translation into English (Latin script) was more successful when the source language also used the Latin script, likely due to more efficient subword tokenization.
  • Multilingual Compression: The researchers successfully trained multilingual students for Baltic and Turkic language families, demonstrating that a single tiny model can handle multiple related languages.

Conclusion

This work demonstrates that high-quality NMT is achievable in resource-constrained environments through strategic knowledge distillation. While teacher quality remains a bottleneck—as seen with the underperforming Japanese model—the results for most languages prove that "tiny" models can bridge the efficiency gap. Future work will focus on expanding the teacher pool and improving cross-script knowledge transfer.

Reference: de Gibert, O., O’Brien, D., Variš, D., & Tiedemann, J. (2025). Mind the Gap: Diverse NMT Models for Resource-Constrained Environments. Proceedings of NoDaLiDa/Baltic-HLT 2025.

Summary updated

Abstract

We present fast Neural Machine Translation models for 17 diverse languages, developed using Sequence-level Knowledge Distillation. Our selected languages span multiple language families and scripts, including low-resource languages. The distilled models achieve comparable performance while being 10x times faster than transformer-base and 35x times faster than transformer-big architectures. Our experiments reveal that teacher model quality and capacity strongly influence the distillation success, as well as the language script. We also explore the effectiveness of multilingual students. We release publicly our code and models in our Github repository: anonymised.
BibTeX
@inproceedings{gibert2025mind,
  author = {Ona de Gibert and Dayyán O'Brien and Dusan Varis and Jörg Tiedemann},
  title = {{Mind the Gap: Diverse NMT Models for Resource-Constrained Environments}},
  booktitle = {NoDaLiDa/Baltic-HLT},
  year = {2025},
  url = {https://openreview.net/forum?id=V6T1qRwTIy},
}
Figure