Dataset Collection and Fine-Tuning of NLLB for German--English Translation

Sk Deeraj, Yash Ingle, Pruthwik MishraarXiv 2026 · arXiv

TL;DR

This paper demonstrates how to achieve state-of-the-art domain adaptation for German–English translation using Meta’s NLLB-200 model on a modest 4GB consumer GPU. By applying a rigorous eight-stage filtering pipeline to the OPUS Books dataset, the authors boosted translation quality by +3.4 BLEU points, proving that high-quality, small-scale data (25k pairs) and clever memory optimization can make powerful AI accessible to individual researchers and small teams.

What Problem Does This Paper Address?

Massive multilingual models like Meta’s "No Language Left Behind" (NLLB) are impressive generalists, but they often struggle with the nuances of specific domains—like the flowery prose of literature or the rigid terminology of technical manuals. Traditionally, "fixing" this through fine-tuning required massive datasets and industrial-grade hardware (think A100 clusters).

The authors address two main hurdles:

  1. Data Noise: Raw web-scale corpora are often messy, containing malformed text or misaligned translations that can degrade a model's performance during fine-tuning.
  2. Hardware Barriers: Fine-tuning transformer models typically demands significant VRAM, putting it out of reach for those with consumer-grade hardware (e.g., an NVIDIA RTX 3050).

Methods

The researchers took the NLLB-200-distilled-600M model and applied a two-pronged strategy: systematic data curation and memory-efficient training.

The Curation Pipeline

Rather than throwing more data at the problem, they focused on better data. They developed an eight-stage pipeline to filter 45,000 raw sentence pairs down to 25,000 "gold-standard" examples. This included:

  • Encoding & Length Filtering: Removing corrupted characters and fragments (like headlines or author names).
  • Language Identification: Using the langdetect library to ensure the German was actually German and the English was English.
  • Fuzzy Deduplication: Removing near-duplicates that differ only by punctuation.
  • Quality Scoring: Discarding the bottom 10% of pairs based on character overlap and perplexity scores.

Memory Optimization

To fit the training process onto a 4GB GPU, they utilized several "pro-level" engineering tricks:

  • Mixed-Precision (FP16): Reducing the memory footprint of weights and gradients by nearly 50%.
  • Gradient Accumulation: Simulating a larger batch size (32) by accumulating gradients over multiple steps before updating the model.
  • Gradient Checkpointing: Trading a bit of speed for memory by recomputing activations during the backward pass instead of storing them.
Table 2: Training hyperparameters and practical settings

Datasets

The study utilized the OPUS Books subcorpus, which contains professionally translated literary works. After the filtering process, the final dataset consisted of 25,000 high-quality German–English parallel sentences.

Table 1: OPUS-derived dataset statistics after filtering

Key Insights and Ideas

  • Quality Trumps Quantity: The "aha moment" is that a carefully curated 25k dataset can yield substantial gains (+3.4 BLEU). In domain adaptation, a clean, small dataset is often more effective than a massive, noisy one.
  • Democratizing AI: By optimizing for 4GB VRAM, the authors show that meaningful NLP research doesn't require a "compute moat." This makes domain-specific translation accessible for offline, privacy-sensitive environments.
  • Stratified Splitting Matters: By ensuring the validation and test sets covered the same range of sentence lengths as the training set, the authors ensured their evaluation was robust across different levels of linguistic complexity.

Key Results

The fine-tuned model showed statistically significant improvements across all major translation metrics:

  • BLEU Score: Increased from 32.4 to 35.8 (+3.4). For context, the authors note this moves the model from "usable but needs heavy editing" to "good first draft" quality.
  • Error Reduction: A manual analysis of 100 samples showed that total errors dropped from 64 to 41. Most notably, mistranslations were halved, and awkward phrasing was reduced by 33%.
Table 3: Translation quality on held-out test set (2,500 pairs)

The model also performed consistently across different sentence lengths, with the most significant gains seen in medium-length sentences (16–40 tokens), which are typical for literary prose.

Table 4: BLEU scores by sentence length category

Limitations and Future Work

While the model is much improved, it still faces challenges inherent to sentence-level translation:

  • Context Blindness: The model sometimes struggles with pronoun resolution (e.g., "he" vs "it") because it doesn't "see" the previous sentence.
  • Rare Terminology: Some highly specialized technical terms were still missed, as they weren't prevalent in the literary-focused training set.

Future Directions: The authors suggest exploring LoRA (Low-Rank Adaptation) for even more efficient parameter sharing, integrating terminology glossaries directly into the tokenizer, and moving toward document-level context modeling to fix those pesky pronoun errors.

Summary updated

Abstract

This paper describes the curation of a high-quality parallel corpus from OPUS Books and the fine-tuning of Meta's NLLB-200 models for German--English translation. We detail a dataset extraction pipeline that filters 45k raw OPUS sentence pairs to 25k high-quality examples through encoding validation, length filtering, language identification, deduplication, and quality scoring. We present a reproducible fine-tuning methodology optimized for consumer GPUs using mixed precision, gradient accumulation, and gradient checkpointing. Our fine-tuned NLLB-600M model achieves statistically significant improvements (+3.4 BLEU, +2.6 chrF++, -3.2 TER) over the base model on literary and technical German-English translation. We demonstrate practical deployment by integrating the fine-tuned model into LinguaHaru, an existing open-source document translation system, achieving format-preserving translation across PDF and Office documents. All dataset preparation scripts, training code, and evaluation tools are provided for reproducibility. The fine-tuned model is available at HuggingFace.
BibTeX
@misc{deeraj2026dataset,
  author = {SK Deeraj and Yash Ingle and Pruthwik Mishra},
  title = {{Dataset Collection and Fine-Tuning of NLLB for German--English Translation}},
  year = {2026},
  eprint = {2611.07797},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.07797},
}
Figure