Abstract
Large language models (LLMs) and their safety classifiers often perform poorly on low-resource languages due to limited training data and evaluation benchmarks. This paper introduces RabakBench, a new multilingual safety benchmark localized to Singapore's unique linguistic context, covering Singlish, Chinese, Malay, and Tamil. RabakBench is constructed through a scalable three-stage pipeline: (i) Generate - adversarial example generation by augmenting real Singlish web content with LLM-driven red teaming; (ii) Label - semi-automated multi-label safety annotation using majority-voted LLM labelers aligned with human judgments; and (iii) Translate - high-fidelity translation preserving linguistic nuance and toxicity across languages. The final dataset comprises over 5,000 safety-labeled examples across four languages and six fine-grained safety categories with severity levels. Evaluations of 11 popular open-source and closed-source guardrail classifiers reveal significant performance degradation. RabakBench not only enables robust safety evaluation in Southeast Asian multilingual settings but also offers a reproducible framework for building localized safety datasets in low-resource environments.
Introduction: The Localization Blind Spot
While Large Language Models (LLMs) have become impressively polyglot, their safety guardrails—the mechanisms designed to prevent the generation of harmful content—remain largely optimized for Western, standard-English contexts. This creates a critical "localization blind spot": guardrails trained on formal linguistic norms often fail to detect harm in regional dialects, slang, and code-mixed speech.
In Singapore, the linguistic landscape is defined by a fluid mix of English (Singlish), Chinese, Malay, and Tamil. Systems that do not understand this context may fail in two ways: they might miss genuine threats expressed in local vernacular, or they might "over-censor" benign cultural expressions, flagging harmless slang as toxic. RabakBench (named after the Singlish word rabak, meaning extreme or outlandish) was developed to address these gaps.
Methodology: The Generate-Label-Translate Pipeline
To construct a high-quality, localized benchmark without the prohibitive cost of purely manual labeling, the researchers developed a scalable three-stage pipeline.
1. Generate
The process begins by transforming raw Singlish web comments into instructional forms. To find challenging "edge cases," the authors used an automated red-teaming loop. An Attack LLM generates candidate Singlish prompts designed to trick a guardrail, while a Critic LLM analyzes failures to refine subsequent attacks. This ensures the benchmark targets the specific weaknesses of current moderation systems.
2. Label
To scale annotation, LLMs were used as primary labelers. To ensure accuracy, the researchers conducted an "Alt-Test" to statistically verify which LLMs most closely aligned with human expert judgments. Only those models with high human-concurrence were allowed to participate in a majority-voting system to assign safety labels across six categories: Hateful, Sexual, Self-Harm, Insults, Physical Violence, and Misconduct.
3. Translate
Standard machine translation often "sanitizes" toxic content, stripping away the harmful intent. RabakBench utilizes a toxicity-preserving translation setup. By using few-shot prompting curated by native language experts, the pipeline ensures that the harmful intent and emotional intensity of the original Singlish are preserved across Chinese, Malay, and Tamil versions.
The RabakBench Dataset
The resulting benchmark, RABAKBENCH, consists of 5,364 parallel entries. Unlike massive, noisier datasets, RABAKBENCH prioritizes human-verified edge cases. A key finding during construction was the "Intent Gap": models frequently struggle with pragmatic intent. For instance, the Singlish phrase "jump down MRT track" is often used as a hyperbolic expression of frustration rather than a literal threat of self-harm. Human annotators recognize this nuance, whereas most automated guardrails do not, leading to unnecessary censorship.
Experimental Results and Discussion
The evaluation of 13 state-of-the-art guardrails revealed a massive performance gap when moving from standard English to localized vernaculars:
- Performance Crater: Models that typically achieve ~86% F1 scores on standard English benchmarks saw their performance drop to as low as 45-47% on RABAKBENCH.
- Commercial Limitations: The Perspective API, a widely used industry standard, achieved an average F1 score of only 28.40%, failing to detect the vast majority of localized harms.
- The Tamil Bottleneck: Across nearly all tested models, performance was significantly lower in Tamil than in other languages, highlighting a severe lack of safety alignment for specific scripts.
- Top Performers: Interestingly, some open-source, safety-tuned models like gpt-oss-safeguard (20b) and Qwen3Guard (8b) proved most robust, maintaining F1 scores above 80% on average, though they still struggled with specific categories in Tamil.
Conclusion
RabakBench demonstrates that current LLM safety systems are not yet ready for the linguistic complexity of global, multilingual societies. The significant degradation in performance across localized dialects and low-resource languages suggests that "one-size-fits-all" safety policies are insufficient. The authors advocate for a shift toward native localized training and the development of context-aware adapters that can navigate regional nuances without sacrificing safety or over-censoring benign cultural speech.
Summary updated