The rapid expansion of Large Language Models (LLMs) into non-English languages has exposed a critical bottleneck: the "silver-standard" data problem. While we can easily use Neural Machine Translation (NMT) to flip English benchmarks into 100 other languages, the resulting text is often riddled with "translationese"—unnatural phrasing, literal but incorrect idioms, and a total loss of cultural context.
To solve this, researchers have converged on a shared strategy: the multi-stage, human-in-the-loop pipeline. Instead of treating translation as a one-step process, these frameworks treat it as an iterative refinement cycle that combines the scale of automation with the nuance of native-speaker expertise.
The Failure of Automated Translation
Standard NMT systems are excellent at preserving the general gist of a sentence but struggle with the "linguistic common sense" required for rigorous evaluation. For example, in SLING: Sino Linguistic Evaluation of Large Language Models, Yixiao Song and colleagues found that even advanced models struggle with hierarchical grammar and social biases in Chinese that native speakers find trivial.
When benchmarks are translated automatically, they often inherit "Western-centric" biases. Shivalika Singh and the team behind Global MMLU discovered that nearly 30% of the standard MMLU benchmark is culturally biased toward the West. If a model fails a question about a specific U.S. legal concept translated into Yoruba, is it failing because it lacks "intelligence" or because the question is culturally irrelevant?
The Three-Stage Refinement Architecture
To bridge this gap, researchers like Ge Zhang 0009 in the Chinese Open Instruction Generalist (COIG) project have formalized a three-stage pipeline:
- Automated Generation: NMT or LLMs perform the initial bulk translation of instructions or questions.
- Expert Verification: Native speakers or domain experts audit the output for "usability" and "naturalness."
- Manual Correction: Experts fix factual errors, cultural mismatches, and grammatical artifacts that automated filters miss.
This architecture ensures that the final dataset is a "gold standard." For instance, in Myanmar XNLI, Aung Kyaw Htet demonstrated that expert-led revision of community-sourced translations significantly improved model performance by correcting subtle polysemy—words with multiple meanings that NMT often confuses.
Scaling Quality Over Quantity
The intuition behind these pipelines is that for low-resource languages, quality beats volume. In the BigScience ROOTS Corpus, Hugo Laurençon and the BigScience team moved away from "scrape-it-all" approaches. They used native speakers to tune language-specific thresholds for quality filters, ensuring that valid text in structurally diverse languages wasn't accidentally deleted by English-centric algorithms.
Similarly, for highly specialized domains like medicine, the TICO-19 initiative by Marcello Federico and others used a two-step vetting process. Professional translations were subjected to a secondary round of vetting by domain experts to ensure that life-saving health information didn't lose its precision during the translation process.
Bridging the Cultural Gap
The most sophisticated versions of these pipelines now include "cultural auditing." In Multi3Hate, Minh Duc Bui used native speakers to audit machine-translated captions of memes. This revealed a "cultural gap" where hate speech detection models, trained on U.S. norms, failed to recognize offensive content in other cultural contexts.
By involving professional translators and language coordinators—as seen in IrokoBench for African languages by David Ifeoluwa Adelani—researchers are finally creating benchmarks that reflect how language is actually used on the ground. These multi-stage pipelines transform translation from a mechanical swap of words into a rigorous process of cultural and linguistic adaptation.