Multilingual Data Filtering using Synthetic Data from Large Language Models

Jonas Waldendorf, Barry Haddow, Alexandra Birch, Mateusz KlimaszewskiEMNLP 2025

TL;DR

This paper introduces Multilingual Data Filtering using Synthetic Data (MDFS), a method that leverages Large Language Models (LLMs) to generate synthetic labels for filtering training data in Neural Machine Translation (NMT). MDFS extends effective English-centric filtering techniques to non-English languages, demonstrating comparable or improved performance in both translation quality and domain-specific filtering tasks.

What Problem Does This Paper Address?

The performance of machine learning models, particularly in Natural Language Processing (NLP), is heavily reliant on the quality and relevance of their training data. While filtering techniques have been widely adopted to improve model performance, most successful approaches have been developed and validated primarily for English. This leaves a significant gap for non-English languages, where data scarcity and quality issues are often more pronounced. The paper addresses this by proposing and evaluating a method to extend effective data filtering to a wider range of languages, including those not officially supported by the LLM used for generating synthetic labels.

Methods

The core of the paper is the MDFS pipeline, which involves three stages:

  1. Synthetic Label Generation: An LLM (specifically Llama-3.1-70B-Instruct) is used to score approximately 500,000 sentences based on predefined criteria (e.g., translation quality, medical domain relevance). This scoring uses an additive 5-point prompt system.
  2. Filtering Model Training: The synthetic labels generated by the LLM are used to train smaller, efficient filtering models. The paper experiments with finetuning XLMR (a pre-trained encoder-only model) using either a classification or regression objective. An alternative, n-gram based approach is also explored for domain-specific filtering.
  3. Data Filtering and NMT Training: The trained MDFS filtering models are then used to score large NMT training corpora. The highest-scoring sentences are selected based on various thresholds to form the final NMT training dataset.

Datasets

The paper utilizes several datasets for its experiments:

  • ParaCrawl: Used for En→De translation quality experiments.
  • CCMatrix: Used for En→Ar translation quality and medical domain experiments.
  • UNPC, HLPT, MultiUN, Neulab-TedTalks: Used in conjunction with CCMatrix for En→Ar medical domain experiments.
  • ELRC Wikipedia-Health: Used specifically for MDFS training data in En→Ar medical domain experiments.
  • ELRC-EMEA: Used for En→Ro medical domain experiments.
  • FLORES-200: Used as the test set for translation quality experiments.
  • TICO-19: Used as the test set for En→Ar medical domain experiments.
  • HIML and WMT18 Biomedical: Combined to form the test set for En→Ro medical domain experiments.

Key Insights and Ideas

  • LLMs as Universal Labelers: The study demonstrates that LLMs can effectively generate synthetic labels for data filtering even for languages they don't officially support, opening up possibilities for low-resource languages.
  • MDFS for Domain Filtering: Beyond just quality, MDFS proves effective in filtering for specific domains (like medical text), outperforming simple keyword-based approaches in some cases.
  • Efficiency of Smaller Models: Training smaller, finetuned models (like XLMR-based ones) on LLM-generated labels is a computationally feasible way to apply sophisticated filtering to massive datasets.
  • Trade-offs in Filtering: While MDFS models can effectively distinguish between "good" and "bad" examples, they struggle to accurately rank the very best (e.g., "excellent") examples, especially at higher thresholds.

Key Results

  • Translation Quality: MDFS models achieved comparable or improved chrF++ scores compared to baselines like RANDOM and COMET-KIWI across various thresholds for En→De and En→Ar translation tasks. For En→De, MDFS-REGRESSION showed a 1.4 chrF++ improvement over the best RANDOM result at a 25% data threshold.
  • Medical Domain Filtering: MDFS models generally outperformed RANDOM and KEYWORD baselines in medical domain filtering for En→Ar and En→Ro. For En→Ar, MDFS-NGRAM was the strongest model, achieving 0.4 chrF++ higher than other MDFS models at thresholds above 1M sentences.
  • Terminology Translation: In domain-specific terminology evaluation, MDFS-CLASS (English) achieved significantly higher win rates (54%) compared to the random baseline, demonstrating its ability to capture domain nuances.
  • LLM Support for Non-English: The study found that MDFS models predicted a greater number of medical sentences when using non-English language inputs compared to English, suggesting LLMs can adapt to domain-specific language patterns.

Limitations and Future Work

  • LLM Spelling Errors: The prompt used for translation quality experiments contained minor spelling mistakes, which could have influenced the LLM's scoring.
  • Bias Reinforcement: Filtering training data can inadvertently reinforce existing biases in the data, and this risk is amplified if the LLM itself has a "Western" bias due to its predominantly English training data.
  • Limited Test Sets: For non-English medical domain experiments, the available test sets were sometimes more general than specific scientific medical writing, potentially affecting results.
  • Future Work: The authors suggest extending this approach to large-scale LLM pre-training datasets (like FineWeb 2) and evaluating filtering on larger, finetuned multilingual models. Further research could also explore different LLMs and labeling strategies.

Summary updated

BibTeX
@inproceedings{waldendorf2025multilingual,
  author = {Jonas Waldendorf and Barry Haddow and Alexandra Birch and Mateusz Klimaszewski},
  title = {{Multilingual Data Filtering using Synthetic Data from Large Language Models}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
}
Figure