BOUQuET : dataset, Benchmark and Open initiative for Universal Quality Evaluation in Translation

Pierre Andrews, Mikel Artetxe, Mariano Coria Meglioli, Marta R. Costa‐jussà, Joe Chuang, David C. Dale, Mark Duppenthaler, Nathanial Paul Ekberg, Cynthia Gao, Daniel J. Licht, Jean Maillard, Alexandre Mourachko, Christophe Ropers, Safiyyah Saleem, Eduardo Sánchez, Ioannis Tsiamas, Arina Turkatenko, Albert Ventayol-Boada, S. J. C. Yates2025 · DOI

Abstract

Machine Translation (MT) evaluation has long been hindered by "English-centric" biases, data contamination, and narrow domain coverage. This paper introduces BOUQuET (Benchmark and Open initiative for Universal Quality Evaluation in Translation), a handcrafted dataset designed to provide a more authentic and context-aware measure of translation quality. By employing native speakers to compose original content across eight major non-English pivot languages, BOUQuET offers a contamination-free benchmark that captures linguistic nuances and diverse registers often overlooked by traditional web-crawled datasets. Initial evaluations of 14 modern systems, including Llama 3 and Aya, reveal that BOUQuET provides a unique signal for model performance that does not perfectly correlate with existing benchmarks like FLORES+.

Introduction

Most modern machine translation benchmarks suffer from three systemic flaws that limit their effectiveness in evaluating state-of-the-art Large Language Models (LLMs). First is English-Centricity: most datasets originate in English and are translated outward, ignoring linguistic features unique to other language families. Second is Data Contamination: as benchmarks are often mined from the web, there is a high risk that LLMs have encountered the test data during training, leading to inflated performance scores. Finally, existing datasets often suffer from a Narrow Scope, focusing heavily on specific domains like news or Wikipedia while lacking variety in "register"—the social context and formality of communication.

BOUQuET addresses these challenges by moving away from automated mining toward manual, multicentric composition. It provides a framework for evaluating translation quality that is culturally grounded and linguistically diverse.

Methodology

Multicentric Design

To break the English-centric mold, BOUQuET utilizes a multicentric approach. Native speakers of eight "pivot" languages—Egyptian Arabic/MSA, French, German, Hindi, Indonesian, Mandarin, Russian, and Spanish—wrote original text from scratch. This ensures the benchmark reflects the natural thought patterns and linguistic structures of each language rather than being a reflection of English-style communication.

Paragraph-Level Context and Linguistic Targeting

Unlike many benchmarks that evaluate isolated sentences, BOUQuET is organized into paragraphs of three to six sentences. This structure allows researchers to test "co-referencing"—the ability of a model to maintain consistency for pronouns and entities across a narrative. Furthermore, linguists were instructed to include specific challenging phenomena, such as pro-drop (omitted subjects), complex case markings, and varying verb moods.

Register Modeling

The researchers defined three functional dimensions to capture the variety of human speech, known as "register":

  1. Connectedness: The relationship between the speakers.
  2. Preparedness: The level of planning or spontaneity in the text.
  3. Social Differential: The power dynamic or formality between participants.

Dataset Composition

The BOUQuET project consists of two primary components:

  • Source-BOUQuET: The core dataset featuring 2,000 sentences across nine languages (the eight pivots plus English).
  • Full-BOUQuET: An ongoing open initiative that has expanded to over 55 languages through community contributions and commissioned translations.

To ensure the benchmark remains "contamination-free," the authors composed the text in 2024 and 2025, ensuring current models have not seen the data. Additionally, a portion of the data remains hidden from the public to prevent future memorization.

Key Insights and Results

Domain Diversity

Using Wasserstein distance to compare probability distributions, the authors demonstrated that BOUQuET covers a significantly wider range of domains—including social media, creative narration, and "how-to" guides—than predecessors like FLORES+ or NTREX. This diversity highlights that "register" is a hidden challenge; model performance varies significantly depending on whether the text is an official speech or a casual social media comment.

Model Performance

In evaluations of 14 systems, Llama 3.3-70B and Aya-Expanse-8B emerged as the top performers. Interestingly, models generally scored higher on BOUQuET than on FLORES+. This suggests that while BOUQuET is linguistically complex, its handcrafted nature makes the text more natural and "easier" to translate than the often clunky, mined text found in other benchmarks.

A New Signal for Evaluation

One of the most significant findings was the low correlation between system rankings on BOUQuET and other datasets. This confirms that BOUQuET is providing a new signal, testing capabilities and linguistic nuances that traditional benchmarks overlook. It also suggests that English is not always the optimal pivot; for certain language pairs, such as Guarani-Spanish, translating through a culturally and linguistically closer pivot is more effective.

Conclusion and Future Work

BOUQuET represents a shift toward more intentional, human-centric evaluation in machine translation. While the initial 2,000-sentence set is a strong foundation, the project's long-term success relies on the Open Initiative to expand language coverage. Future research will focus on granular linguistic analysis to identify specific grammatical structures—such as gender agreement or complex verb tenses—that continue to pose challenges for even the most advanced LLMs.

Summary updated

BibTeX
@misc{andrews2025bouquet,
  author = {Pierre Andrews and Mikel Artetxe and Mariano Coria Meglioli and Marta R. Costa‐jussà and Joe Chuang and David C. Dale and Mark Duppenthaler and Nathanial Paul Ekberg and Cynthia Gao and Daniel J. Licht and Jean Maillard and Alexandre Mourachko and Christophe Ropers and Safiyyah Saleem and Eduardo Sánchez and Ioannis Tsiamas and Arina Turkatenko and Albert Ventayol-Boada and S. J. C. Yates},
  title = {{BOUQuET : dataset, Benchmark and Open initiative for Universal Quality Evaluation in Translation}},
  year = {2025},
  doi = {10.18653/v1/2025.emnlp-main.1400},
}
Figure