Lost in Localization: Building RabakBench with Human-in-the-Loop Validation to Measure Multilingual Safety Gaps

Gabriel Yong Ping Chua, Lijia Tan, Ziyu Ge, Roy Ka-Wei LeearXiv 2025 · DOI · arXiv

Abstract

Large language models (LLMs) often fail to maintain safety in low-resource language varieties, such as code-mixed vernaculars and regional dialects. We introduce RabakBench, a multilingual safety benchmark and scalable pipeline localized to Singapore's unique linguistic landscape, covering Singlish, Chinese, Malay, and Tamil. We construct the benchmark through a three-stage pipeline: (1) Generate: augmenting real-world unsafe web content via LLM-driven red teaming; (2) Label: applying semi-automated multi-label annotation using majority-voted LLM labelers; and (3) Translate: performing high-fidelity, toxicity-preserving translation. The resulting dataset contains over 5,000 examples across six fine-grained safety categories. Despite using LLMs for scalability, our framework maintains rigorous human oversight, achieving 0.70-0.80 inter-annotator agreement. Evaluations of 13 state-of-the-art guardrails reveal significant performance degradation, underscoring the need for localized evaluation. RabakBench provides a reproducible framework for building safety benchmarks in underserved communities.

Introduction: The Localization Blind Spot

Most Large Language Model (LLM) safety research focuses on standard varieties of high-resource languages. However, in real-world settings, communication often occurs in "vernaculars"—complex blends of languages, regional dialects, and culturally specific slang. Singapore serves as a critical stress test for these systems, as its residents fluidly navigate between English, Mandarin, Malay, Tamil, and Singlish (an English-based creole).

Current safety guardrails frequently exhibit a "localization blind spot." They may fail to identify localized slurs while simultaneously "over-censoring" benign cultural expressions. For example, a model might flag a common Singlish expression of frustration as a high-severity self-harm threat because it lacks the cultural context to understand the pragmatic intent behind the words.

An example of a localized adversarial prompt targeting Singlish-specific cultural contexts and derogatory terms.

Methodology: The Generate-Label-Translate Pipeline

To address these gaps, we developed a reproducible three-stage pipeline to build a high-quality, localized safety dataset:

The three-stage RabakBench pipeline: (1) Generate via red-teaming, (2) Label via validated LLM ensembles, and (3) Translate via toxicity-preserving methods.

1. Generate

We curated real-world Singlish comments from web forums and utilized an iterative "Attack LLM/Critic LLM" loop. This adversarial approach red-teams existing guardrails to find examples they missed, ensuring the benchmark captures challenging, real-world edge cases.

2. Label

To scale annotation without losing quality, we employed the Alt-Test—a statistical method to identify which LLMs most closely align with human expert judgments. We selected a "jury" of high-performing LLMs (including Gemini 2.0 Flash and o3-mini) to assign fine-grained safety labels via majority vote, achieving inter-annotator agreement levels comparable to human experts.

3. Translate

We extended the dataset into Chinese, Malay, and Tamil using a toxicity-preserving translation setup. Standard translation often "sanitizes" harmful text; our pipeline uses few-shot prompting and human expert workshops to ensure translated versions maintain the same level of "heat" and cultural nuance as the original Singlish source.

Dataset Characteristics

Named after the Singlish word rabak (meaning "extreme" or "intense"), RABAKBENCH consists of 5,364 entries. The dataset uses a taxonomy of six fine-grained categories (Hateful, Sexual, Self-Harm, Insults, Physical Violence, and Misconduct) with two levels of severity. While smaller than massive general datasets, its specialized nature makes it a significantly more rigorous test for regional safety.

Key Results and Insights

Evaluations of 13 state-of-the-art guardrails, including commercial tools like the Perspective API and OpenAI Moderation, revealed sobering results:

  • Significant Performance Degradation: Models that scored ~80% F1 on standard English benchmarks saw their scores plummet to the 40-50% range on RABAKBENCH.
  • The Tamil Floor: Performance in Tamil was consistently the lowest across almost all models, with several scoring below 30% F1. This highlights a critical deficiency in how current safety systems handle specific scripts and regional contexts.
  • The Intent Gap: Models struggle with "hyperbolic" localized speech. Phrases used in Singlish to express mere annoyance are frequently misclassified as literal threats, indicating that "global" safety training is failing to generalize to underserved communities.

Conclusion

The persistent performance gaps identified by RabakBench suggest that localized evaluation is essential for safe LLM deployment. Our "Generate-Label-Translate" blueprint provides a scalable framework for other researchers to build similar benchmarks for diverse linguistic regions worldwide.

Summary updated

Abstract

Large language models (LLMs) often fail to maintain safety in low-resource language varieties, such as code-mixed vernaculars and regional dialects. We introduce RabakBench, a multilingual safety benchmark and scalable pipeline localized to Singapore's unique linguistic landscape, covering Singlish, Chinese, Malay, and Tamil. We construct the benchmark through a three-stage pipeline: (1) Generate: augmenting real-world unsafe web content via LLM-driven red teaming; (2) Label: applying semi-automated multi-label annotation using majority-voted LLM labelers; and (3) Translate: performing high-fidelity, toxicity-preserving translation. The resulting dataset contains over 5,000 examples across six fine-grained safety categories. Despite using LLMs for scalability, our framework maintains rigorous human oversight, achieving 0.70-0.80 inter-annotator agreement. Evaluations of 13 state-of-the-art guardrails reveal significant performance degradation, underscoring the need for localized evaluation. RabakBench provides a reproducible framework for building safety benchmarks in underserved communities.
BibTeX
@misc{chua2025lost,
  author = {Gabriel Yong Ping Chua and Lijia Tan and Ziyu Ge and Roy Ka-Wei Lee},
  title = {{Lost in Localization: Building RabakBench with Human-in-the-Loop Validation to Measure Multilingual Safety Gaps}},
  year = {2025},
  eprint = {2507.05980},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2507.05980},
  howpublished = {arXiv},
}
Figure