Toxicity-Aware Few-Shot Prompting for Low-Resource Singlish Translation

Ziyu Ge, Gabriel Chua, Leanne Tan, Roy Ka-Wei LeearXiv 2025 · DOI · arXiv

Introduction

Translating informal, code-mixed languages like Singlish—a Singaporean creole blending English, Malay, Hokkien, and other regional dialects—presents a unique challenge for standard machine translation systems. Beyond the lack of large parallel datasets, modern Large Language Models (LLMs) often employ aggressive safety filters that "sanitize" or "clean up" offensive input. While intended to prevent harm, this behavior is counterproductive for tasks like content moderation or regional platform governance, where accurately identifying and translating local insults is essential. This research introduces a two-stage, human-in-the-loop framework designed to preserve sociolinguistic nuance and toxicity in low-resource Singlish translation into Chinese (ZH), Malay (MS), and Tamil (TA).

Methodology

The proposed framework addresses the limitations of standard LLM translation through iterative human curation and automated prompt optimization.

Stage 1: Human-Curated Few-Shot Example Pool

To guide the LLM, the researchers developed a three-round human verification process to curate 20 "gold standard" translation pairs, balanced between benign and harmful content. In the first round, LLMs generated candidates which humans then filtered or rewrote. The second round involved ranking these candidates, and the final round selected the single best translation for the few-shot prompt pool.

Stage 2: Model Selection & Prompt Optimization

The researchers evaluated four LLMs: GPT-4o mini, Gemini 2.0 Flash, Grok 3 Beta Mini, and DeepSeek-R1. Evaluation relied on two automated proxies: Direct Semantic Similarity (comparing source Singlish to target translation) and Back-Translation Similarity (translating the target back to Singlish and comparing it to the original). Dynamic prompting was used to select the top-k most semantically similar examples from the curated pool to guide the model for each specific input.

The two-stage framework for curating few-shot examples and optimizing model selection.

Experimental Results

Model Performance and the "Tamil Gap"

Quantitative analysis revealed significant performance variance across models and target languages. GPT-4o mini emerged as the most effective model for this task, consistently achieving the highest semantic fidelity. It particularly excelled in retaining tone and toxicity without over-sanitization, often matching or surpassing human-translated baselines for Chinese and Malay.

Semantic similarity metrics comparing various LLMs across Chinese (ZH), Malay (MS), and Tamil (TA).

As shown in the results, Tamil (TA) proved to be the most challenging target language. Scores for Tamil were significantly lower across all models (e.g., GPT-4o mini scored 29.50 in direct translation for TA compared to 69.50 for ZH). Annotators noted that Tamil outputs were often "emotionally flat" or substituted insults with polite forms, likely due to a lack of informal Tamil training data or stricter safety tuning for that language.

Linguistic Insights

  • Orthographic Variability: Malay presented challenges for automation; annotators frequently preferred custom translations with colloquial spellings to better capture the "vibe" of the original Singlish.
  • Linguistic Distance: Because Singlish already incorporates many Malay and Chinese loanwords, translations into these languages felt more natural to the models than translations into the morphosyntactically distant Tamil.

Conclusion and Future Work

The iterative human-in-the-loop approach successfully converged on high-quality examples, as evidenced by improving Jaccard agreement between annotators across curation rounds. However, limitations remain. The framework may still miss implicit toxicity or culturally specific sarcasm that lacks explicit "bad words." Future work will focus on improving performance for distant languages like Tamil and refining metrics to better capture implicit harm. The study highlights that "toxicity" remains a moving target, influenced by the demographic and cultural background of the annotators.

Summary updated

Abstract

As online communication increasingly incorporates under-represented languages and colloquial dialects, standard translation systems often fail to preserve local slang, code-mixing, and culturally embedded markers of harmful speech. Translating toxic content between low-resource language pairs poses additional challenges due to scarce parallel data and safety filters that sanitize offensive expressions. In this work, we propose a reproducible, two-stage framework for toxicity-preserving translation, demonstrated on a code-mixed Singlish safety corpus. First, we perform human-verified few-shot prompt engineering: we iteratively curate and rank annotator-selected Singlish-target examples to capture nuanced slang, tone, and toxicity. Second, we optimize model-prompt pairs by benchmarking several large language models using semantic similarity via direct and back-translation. Quantitative human evaluation confirms the effectiveness and efficiency of our pipeline. Beyond improving translation quality, our framework contributes to the safety of multicultural LLMs by supporting culturally sensitive moderation and benchmarking in low-resource contexts. By positioning Singlish as a testbed for inclusive NLP, we underscore the importance of preserving sociolinguistic nuance in real-world applications such as content moderation and regional platform governance.
BibTeX
@misc{ge2025toxicity,
  author = {Ziyu Ge and Gabriel Chua and Leanne Tan and Roy Ka-Wei Lee},
  title = {{Toxicity-Aware Few-Shot Prompting for Low-Resource Singlish Translation}},
  year = {2025},
  eprint = {2507.11966},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2507.11966},
}
Figure