The Art of Asking: Multilingual Prompt Optimization for Synthetic Data

David Mora, Viraat Aryabumi, Wei-Yin Ko, Sara Hooker, Julia Kreutzer, Marzieh FadaeearXiv 2025 · arXiv · OpenReview

TL;DR

This paper introduces a novel "prompt-focused" paradigm for generating synthetic data for multilingual Large Language Models (LLMs). Instead of just optimizing generated text, they systematically transform translated prompts to be more natural, culturally relevant, and challenging. This approach significantly boosts LLM performance across various benchmarks, demonstrating a more effective way to build globally capable and culturally aware models.

What Problem Does This Paper Address?

Scaling LLMs, especially for multilingual tasks, often relies on synthetic data. A common method is to translate English prompts into target languages. However, this "translation-based" approach has a major drawback: it inherits the English-centric framing and style, neglecting crucial cultural nuances. This leads to models that are less generalizable and may even perpetuate biases. The authors argue that the prompt itself, the input that guides the LLM, is an underutilized lever for improving multilingual performance. They aim to address the limitations of translation-only prompts by directly optimizing the prompt distribution to be more aligned with real-world language use across diverse cultures.

Methods

The core of this paper is a "prompt-focused paradigm" that reframes synthetic data generation. Instead of solely optimizing the output (completions), they focus on optimizing the input (prompts). Their lightweight framework involves systematically transforming translated prompts along three key dimensions:

  • Naturalness (Tnat): This transformation aims to remove translation artifacts and restore idiomatic phrasing, making the prompts sound more authentic to native speakers.
  • Cultural Adaptation (Tcult): This step recontextualizes prompts with locally relevant examples, values, and references to align them with cultural norms.
  • Difficulty Enhancement (Tdiff): This transformation increases the complexity of prompts by expanding or reformulating them into more challenging, multi-step instructions.

These transformations are applied using an off-the-shelf multilingual LLM. The authors also explore applying these transformations individually or in sequence, with "Naturalness" being applied first, followed by "Cultural Adaptation" and "Difficulty Enhancement" on top of the naturalized prompts.

Datasets

The paper utilizes a combination of existing and implicitly generated datasets:

  • English Seed Prompts: A pool of 280k noisy but real-world English prompts collected from user interactions, filtered for content and language.
  • Translated Prompts: 10k sub-samples of these English prompts were translated into 12 target languages using an in-house translation LLM.
  • Transformed Prompts: The core of their data generation involves applying the "Naturalness," "Cultural Adaptation," and "Difficulty Enhancement" transformations to these translated prompts.
  • Downstream Benchmarks: The transformed data is used to fine-tune LLMs for evaluation on various benchmarks, including:
  • Global-MMLU: For measuring knowledge access across cultures.
  • Flores XCometXL: For machine translation quality.
  • mArenaHard: For open-ended generation quality.
  • PolyWrite: For creative writing and naturalness.
  • Include44: For culturally specific academic and professional exam questions.
  • MGSM: For mathematical reasoning.

Key Insights and Ideas

  • The Prompt Space is a Powerful Lever: The most significant insight is that optimizing the input prompts themselves, rather than just the output completions, offers a more effective path to improving multilingual LLM performance. This challenges the traditional generation-focused paradigm.
  • Beyond Translationese: Simply translating prompts, even with high-quality translation models, is insufficient. The inherent biases and English-centric framing of translated prompts limit model generalization and cultural grounding.
  • Cultural Nuance Matters: Adapting prompts to be culturally relevant significantly boosts performance on tasks requiring cultural understanding, demonstrating that LLMs can benefit from prompts that reflect local contexts.
  • Difficulty Drives Performance: Increasing prompt difficulty, when done thoughtfully, leads to substantial improvements, particularly in areas like mathematical reasoning and machine translation.
  • Synergy of Transformations: Combining different prompt transformations (e.g., Cultural Adaptation and Difficulty Enhancement) often yields the best overall results, suggesting that a well-rounded prompt distribution is key.

Key Results

The authors report substantial and consistent improvements across various benchmarks:

  • Global-MMLU Accuracy: +4.7% improvement over the translation-only baseline.
  • Flores XCometXL: +2.4% improvement.
  • mArenaHard Preferences: +35.3% win rate against the translated baseline.
  • PolyWrite: Significant gains in naturalness and win rates, particularly for creative writing tasks.
  • Lowest-Resource Languages: The prompt optimization approach shows particular effectiveness in improving performance for languages that are less supported by pre-training, demonstrating its utility for language expansion.
  • Completion Length: Difficulty-enhanced prompts led to completions that were, on average, 2.2x longer, providing more training data.

Limitations and Future Work

  • Reliability of Synthetic Data: The authors acknowledge the inherent risks of synthetic data, including potential biases or errors from the teacher model that might not be caught by evaluation metrics. They recommend exploring more targeted filters and involving native speakers for data inspection.
  • Language Scope: While the study covers 12 languages, further work is needed to confirm if the observed performance transfers to similarly positioned languages. Geographic proximity might also play a role.
  • Cold Start Problem: The method relies on bootstrapping from existing models and automatic translation, making it less effective for truly "cold start" scenarios with extremely low-resource languages.
  • Translator Choice: The quality of the initial translation significantly impacts the effectiveness of prompt transformations. Future work could involve hand-picking the best available translator for each language and task.
  • Generation Process Optimization: Further improvements could come from optimizing the generation process itself, potentially using multiple teachers or quality filters.

Summary updated

Abstract

Synthetic data has become a cornerstone for scaling large language models, yet its multilingual use remains bottlenecked by translation-based prompts. This strategy inherits English-centric framing and style and neglects cultural dimensions, ultimately constraining model generalization. We argue that the overlooked prompt space-the very inputs that define training distributions-offers a more powerful lever for improving multilingual performance. We introduce a lightweight framework for prompt-space optimization, where translated prompts are systematically transformed for Naturalness, Cultural Adaptation, and Difficulty Enhancement. Using an off-the-shelf multilingual LLM, we apply these transformations to prompts for 12 languages spanning 7 families. Under identical data conditions, our approaches achieve substantial and consistent downstream improvements over the translation-only baseline: +4.7% on Global-MMLU accuracy, +2.4% on Flores XCometXL and +35.3% wins in preferences on mArenaHard. We establish prompt-space optimization as a simple yet powerful paradigm for building multilingual LLMs that are more robust, culturally grounded, and globally capable.
BibTeX
@misc{mora2025art,
  author = {David Mora and Viraat Aryabumi and Wei-Yin Ko and Sara Hooker and Julia Kreutzer and Marzieh Fadaee},
  title = {{The Art of Asking: Multilingual Prompt Optimization for Synthetic Data}},
  year = {2025},
  eprint = {2510.19806},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2510.19806},
}
Figure