DIETA: A Decoder-only Transformer-based Model for Italian-English Machine TrAnslation

Pranav Kasela, Marco Braga, Alessandro Ghiotto, Andrea Pilzer, Marco Viviani, Alessandro Raganato

Introduction

While massive multilingual Large Language Models (LLMs) like GPT-4 or NLLB-200 have revolutionized natural language processing, they often prioritize breadth over depth. Supporting hundreds of languages simultaneously can lead to performance lags in specific language pairs, such as Italian–English, compared to specialized systems. Furthermore, these giant models are computationally expensive to deploy.

DIETA (Italian for "diet") addresses this gap as a compact, 0.5-billion parameter decoder-only Transformer specifically optimized for high-quality Italian–English translation. By leveraging a massive curated dataset of 768 million sentence pairs, it matches or outperforms models nearly 20x its size, demonstrating that targeted data engineering allows small models to compete with significantly larger baselines.

Methodology

DIETA departs from the traditional encoder-decoder architecture common in Machine Translation (MT) in favor of a decoder-only Transformer architecture, the current standard for modern LLMs. The model consists of 6 layers with a hidden dimension of 2048 and 32 attention heads.

To capture morphological nuances better than general-purpose tokenizers, the authors utilized the SentencePiece vocabulary from the Minerva model family, which was specifically trained on Italian and English text. A critical component of the pipeline was quality control; the authors employed an LLM as a "quality judge" to filter the parallel corpus, ensuring only high-quality translations were used for training.

The filtering prompt used to evaluate translation quality during data curation.

Data Engineering and Curation

The model's performance is rooted in a massive, three-tiered training pipeline:

  1. Parallel Corpus: 207 million high-quality bilingual pairs sourced from Europarl, legal texts, and subtitles.
  2. Synthetic Data: 352 million additional pairs generated via back-translation. Monolingual Italian and English news/web texts were translated back into the source language to create a robust synthetic training set.
  3. WikiNews-25 Benchmark: To evaluate contemporary language use, the authors introduced a new set of 450 sentences from 2025 WikiNews articles, human-post-edited to ensure accuracy.

Key Results and Evaluation

DIETA consistently performed in the second quartile of a 32-system leaderboard, outperforming nearly all other models under 3 billion parameters. On the WikiNews-25 benchmark, DIETA matched the performance of Tower-7B and NLLB-3.3B despite being significantly smaller.

The model delivers competitive translation quality while requiring substantially less GPU memory than top-performing multilingual baselines. The results below illustrate how different DIETA variants (including those with back-translation and synthetic data) compare against established models across various metrics.

Comprehensive performance comparison of DIETA variants against baseline models across multiple translation metrics.

Insights and Future Directions

The success of DIETA highlights that a 0.5B model can rival 7B or 9B parameter models if the training data is sufficiently large and high-quality. The inclusion of synthetic data (the +BT and +allsynth variants) was instrumental in closing the performance gap with larger models, particularly in diverse domains like news.

However, a gap remains in reference-free Quality Estimation (QE), where larger models still maintain an edge in assessing language fluency without a "gold standard" reference. Future work will focus on QE-aware fine-tuning, exploring Sparse Mixture of Experts (MoE) to increase capacity efficiently, and implementing 4-bit or 8-bit quantization for edge deployment on mobile and low-power hardware.

Summary updated

BibTeX
@inproceedings{kasela2025dieta,
  author = {Pranav Kasela and Marco Braga and Alessandro Ghiotto and Andrea Pilzer and Marco Viviani and Alessandro Raganato},
  title = {{DIETA: A Decoder-only Transformer-based Model for Italian-English Machine TrAnslation}},
  booktitle = {Clic-it},
  year = {2025},
}
Figure