RELIC: Enhancing Reward Model Generalization for Low-Resource Indic Languages with Few-Shot Examples

Soumya Suvra Ghosal, Vaibhav Singh, Akash Ghosh, Saheb Pal, Sandip Kumar Baidya, Sriparna Saha, Dinesh Manocha2025 · DOI

Introduction

The alignment of Large Language Models (LLMs) with human values typically relies on Reinforcement Learning from Human Feedback (RLHF). Central to this process is the Reward Model (RM), which is trained to rank model responses based on human preferences. However, a significant "alignment gap" exists for low-resource languages. While high-resource languages like English benefit from vast preference datasets, low-resource languages—particularly Indic languages such as Bodo, Santali, and Manipuri—lack the necessary data for effective RM fine-tuning. This often results in reward models that fail to distinguish between safe and harmful content in these languages, posing a substantial barrier to safe AI deployment globally.

The Challenge of Low-Resource Alignment

Current reward models often exhibit poor generalization when applied to low-resource contexts. In many instances, a model might assign a higher reward to a harmful response than a safe one simply because it lacks the linguistic depth to understand the target language's nuances. Collecting human-labeled preference data for every language is economically and logistically unfeasible. Consequently, researchers must find ways to leverage existing high-resource data to improve model performance in low-resource settings without extensive retraining.

RELIC: A Novel In-Context Learning Framework

To address this, the authors introduce RELIC (REward model generalization for Low-resource Indic languages with few-shot Contexts). RELIC bypasses the need for fine-tuning by utilizing In-Context Learning (ICL). By providing the reward model with carefully selected "hints" or examples during inference, RELIC helps the model calibrate its evaluations for the target language.

The RELIC framework operates in two primary stages:

  1. Auxiliary Dataset Selection: The system identifies high-resource "cousin" languages (e.g., Hindi or Bengali) that share linguistic similarities with the target low-resource language (e.g., Bodo). Preference data from these high-resource languages serves as an "auxiliary bank."
  2. Pairwise Ranking Retriever: Unlike standard retrievers that prioritize semantic similarity to the query, RELIC employs a retriever trained with a pairwise ranking loss. This retriever is specifically designed to select pairs of examples—one preferred and one less-preferred—that best illustrate the distinction between high-quality and low-quality responses.

By prepending these discriminative pairs to the user's prompt, RELIC provides the reward model with a "discriminative context" that clarifies the boundary between desirable and undesirable outputs.

Experimental Setup

The researchers evaluated RELIC across four low-resource Indic languages: Bodo, Santali, Manipuri, and Odia. They utilized three major preference datasets, translated and manually curated for accuracy:

  • PKU-SafeRLHF: Focused on safety and harm reduction.
  • WebGPT: Focused on question-answering quality.
  • HH-RLHF: Focused on helpfulness and harmlessness.

The auxiliary banks were drawn from 10 high-resource Indic languages, including Bengali, Hindi, Marathi, and Tamil.

Key Insights

Similarity vs. Utility

A core finding of the study is that semantic similarity is not the optimal metric for retrieval in reward modeling. While standard approaches look for examples similar to the prompt, RELIC demonstrates that the model benefits more from seeing the contrast between responses. Providing a "good" and "bad" pair in a related language offers a stronger signal for alignment than simply showing a similar safe response.

The "Bodo Gap" and Cross-Lingual Transfer

The study highlights a critical failure in current multilingual models. For example, a LLaMA-3.1-8B safety model in a zero-shot setting actually preferred unsafe responses in Bodo. This suggests that multilingual pre-training does not inherently lead to multilingual value alignment. However, RELIC shows that a model's understanding of safety in a high-resource language like Hindi can be "unlocked" and transferred to a low-resource language if the right context is provided.

Results

RELIC consistently outperformed zero-shot prompting and existing state-of-the-art retrieval methods across all tested languages and datasets.

  • Accuracy Improvements: On the Bodo language using a LLaMA-3.2-3B reward model, RELIC achieved a 12.81% improvement in accuracy over zero-shot baselines and a 10.13% improvement over the previous best retrieval method.
  • Safety Benchmarks: In the PKU-SafeRLHF dataset, RELIC improved accuracy in Santali by over 21% compared to existing methods.
  • Representation Analysis: UMAP visualizations of the model's hidden layers revealed that without RELIC, safe and unsafe responses were often entangled. With the addition of RELIC’s in-context examples, these responses formed distinct, separable clusters, significantly simplifying the model's classification task.

Limitations and Future Directions

While RELIC shows great promise for Indic languages, its effectiveness across diverse language families remains to be explored. Additionally, the current study focuses on pairwise accuracy; future research should investigate how these improved reward signals impact the final performance of LLMs after full RLHF training. Finally, extending RELIC to generative or implicit reward models, such as those used in Direct Preference Optimization (DPO), represents a valuable next step.

Conclusion

RELIC provides a scalable, data-efficient solution for aligning LLMs in low-resource linguistic environments. By leveraging the discriminative power of high-resource language pairs, it bridges the gap between multilingual capability and multilingual safety, ensuring that the benefits of AI alignment are more equitably distributed.

Summary updated

BibTeX
@misc{ghosal2025relic,
  author = {Soumya Suvra Ghosal and Vaibhav Singh and Akash Ghosh and Saheb Pal and Sandip Kumar Baidya and Sriparna Saha and Dinesh Manocha},
  title = {{RELIC: Enhancing Reward Model Generalization for Low-Resource Indic Languages with Few-Shot Examples}},
  year = {2025},
  doi = {10.18653/v1/2025.findings-emnlp.79},
}
Figure