TL;DR
This paper introduces multilingual Process Reward Models (PRMs) to improve multi-step reasoning in Large Language Models (LLMs) across seven languages. They demonstrate that multilingual PRMs significantly outperform monolingual and cross-lingual approaches, leading to better accuracy and fewer early-stage reasoning errors, opening doors for more robust multilingual AI applications.
What Problem Does This Paper Address?
Large Language Models (LLMs) are increasingly capable of complex tasks, but their ability to perform multi-step reasoning, especially in diverse languages, remains a challenge. While Process Reward Models (PRMs) have shown promise in guiding LLMs by providing step-by-step feedback, most existing work focuses solely on English. This creates a significant gap, as LLMs are being deployed in multilingual contexts, and PRMs trained only on English may not generalize well or could even introduce biases. The authors aim to bridge this gap by developing and evaluating multilingual PRMs for complex reasoning tasks.
Methods
The core of this work involves extending Process Reward Modeling (PRM) to a multilingual setting. The authors:
- Translate Existing Datasets: They took two established PRM datasets, PRM800K and Math-Shepherd, originally in English, and translated them into six additional languages (German, Spanish, French, Russian, Swahili, and Chinese) using NLLB 3.3B. This created a multilingual training corpus.
- Define PRM Setups: They established three distinct PRM configurations for comparison:
- PRM-MONO: Trained and evaluated on a single language.
- PRM-CROSS: Trained on English data but evaluated on all languages.
- PRM-MULTI: Trained on all seven translated languages and evaluated on all test languages.
- Evaluate on Reasoning Benchmarks: They used two popular math reasoning benchmarks, MGSM and MATH500, translated into 11 languages (seven seen during training and four unseen).
- Utilize Multiple LLMs: The experiments were conducted using three different LLMs as generators: METAMATH-MISTRAL-7B, LLAMA-3.1-8B-MATH, and DEEPSEEKMATH-7B-INSTRUCT.
- Analyze Key Factors: They investigated the impact of the number of training languages, the proportion of English data, the number of candidate responses, and parameter-efficient finetuning (PEFT) methods like LoRA.
Datasets
- Training Datasets:
- PRM800K (translated from English to 6 additional languages)
- Math-Shepherd (translated from English to 6 additional languages)
- Evaluation Datasets:
- MGSM (translated into 11 languages)
- MATH500 (translated into 11 languages)
The authors carefully filtered translated data to ensure numerical and equation consistency and used human translators and GPT-40 to improve translation quality for both training and evaluation sets.
Key Insights and Ideas
- Multilingual PRMs are Superior: The most significant finding is that PRM-MULTI consistently outperforms both PRM-MONO and PRM-CROSS across all tested LLMs and benchmarks. This highlights the power of diverse language training for reasoning.
- Fewer Early Errors: Multilingual PRMs are less prone to making errors in the initial steps of reasoning, which is crucial for preventing error propagation and improving overall accuracy.
- Language Diversity Matters, But There's a Sweet Spot: While more languages generally improve performance, adding languages beyond five did not yield further gains. This suggests a trade-off where too many languages with fixed data can lead to insufficient training per language.
- English Data Balance is Key: A moderate amount of English data (even as low as 1%) in the training mixture significantly boosts performance, especially for unseen languages. However, too much English data (over 50%) can lead to performance degradation.
- Candidate Responses and Parameters Boost Performance: Multilingual PRMs benefit more from a larger number of candidate responses and trainable parameters, indicating scalability.
Key Results
- Accuracy Improvement: PRM-MULTI boosted average accuracy by up to +1.2 and +1.5 points compared to PRM-CROSS and PRM-MONO, respectively.
- Reduced Early Errors: Visualizations show a clear reduction in the percentage of earliest errors occurring in the first few steps for PRM-MULTI compared to other setups.
- Cross-lingual Transfer: PRM-MULTI demonstrated superior performance on unseen languages, indicating effective generalization.
- Comparison to ORM: Multilingual PRMs consistently outperformed Outcome Reward Models (ORMs) and Self-Consistency methods, achieving significant accuracy gains.
- RL Fine-tuning Benefits: Using multilingual PRMs for Reinforcement Learning (RL) fine-tuning with PPO further improved LLM performance, showing an average boost of +1.86 and +2.19 compared to baseline and PPO-ORM.
Limitations and Future Work
- Limited Language Scope: The study was conducted on 11 languages. Future work should explore a broader range of languages and diverse linguistic families to assess the impact on multilingual PRMs.
- Reward Optimization Methods: The paper focused on best-of-N and PPO. Further research could investigate other reward optimization techniques and their compatibility with cross-lingual reward model transfer.
- Language Similarity: The study found no clear correlation between language similarity (script, family) and cross-lingual transfer performance, suggesting that selecting a diverse set of representative languages might be more effective than relying solely on similarity.
- Checkpoint Selection in RL: The authors noted that the checkpoint selection strategy in RL fine-tuning, based on average loss across all languages, might not yield optimal performance for individual languages, as seen with Bengali. Future work could explore language-specific adjustments.
Summary updated