ToxiCloakCN: Evaluating Robustness of Offensive Language Detection in Chinese with Cloaking Perturbations

Yunze Xiao, Yujia Hu, Kenny Tsu Wei Choo, Roy Ka-Wei Lee

Abstract

Detecting hate speech and offensive language is essential for maintaining a safe and respectful digital environment. This study examines the limitations of state-of-the-art large language models (LLMs) in identifying offensive content within systematically perturbed data, with a focus on Chinese, a language particularly susceptible to such perturbations. We introduce ToxiCloakCN, an enhanced dataset derived from ToxiCN, augmented with homophonic substitutions and emoji transformations, to test the robustness of LLMs against these cloaking perturbations. Our findings reveal that existing models significantly underperform in detecting offensive content when these perturbations are applied. We provide an in-depth analysis of how different types of offensive content are affected by these perturbations and explore the alignment between human and model explanations of offensiveness. Our work highlights the urgent need for more advanced techniques in offensive language detection to combat the evolving tactics used to evade detection mechanisms.

Introduction

In the digital landscape, the moderation of offensive language is a perpetual "arms race" between automated detection systems and users seeking to bypass them. This conflict is particularly pronounced in the Chinese internet ecosystem, where strict lexicon-based censorship has fostered a unique linguistic phenomenon known as cloaking. To evade filters, users frequently replace offensive terms with homophones (characters that share the same sound but differ in meaning) or semantically related emojis.

While human readers can intuitively decode these perturbations by "hearing" the intended insult or interpreting visual subtext, Large Language Models (LLMs) often struggle. This paper introduces ToxiCloakCN, a benchmark designed to quantify the robustness of state-of-the-art models against these evolving evasion tactics.

Methodology

Dataset Construction

The researchers developed ToxiCloakCN by transforming the existing ToxiCN dataset—a collection of offensive language sourced from platforms like Zhihu and Tieba. The new benchmark consists of 4,582 balanced sentences (2,293 offensive, 2,289 non-offensive). Two primary perturbation methods were applied at a 0.3 ratio (replacing 30% of characters) to ensure the text remained human-readable while challenging machine logic:

  1. Homophone Replacement: Keywords were swapped with phonetically identical characters using the JioNLP dictionary. For example, the slur "Henan people" (河南人) might be masked as "He Nanren" (贺楠仁).
  2. Emoji Replacement: Keywords were replaced with visually or semantically related emojis, such as using a dog emoji to represent the derogatory term "simp" (舔狗).

Experimental Setup

The study evaluated a range of models, including the closed-source GPT-4o and fine-tuned open-source models such as LLaMA-3-8B, Qwen1.5-MoE, and Mistral-7B. To explore the impact of prompting, the researchers tested six different instruction templates, varying the language (English vs. Chinese) and the input format (Standard Text vs. Pinyin phonetic romanization).

Key Findings

The Performance Gap

Every model tested exhibited a significant decline in performance when faced with cloaked text. GPT-4o, the most robust model, saw its Macro F1 score drop from 0.796 on the original dataset to 0.727 on homophones and 0.754 on emojis. Open-source models, while competitive on standard text, struggled significantly more with emoji-based perturbations, indicating a gap in their visual-semantic reasoning capabilities.

The Failure of Lexicon-Based Detection

Traditional moderation systems that rely on "blacklists" of offensive words were rendered virtually useless by ToxiCloakCN. Lexicon-based detectors saw their F1 scores plummet to near zero (0.003), as the literal "bad words" were no longer present in the text, even though the offensive intent remained clear to a human reader.

The Pinyin Paradox

A key hypothesis of the study was that providing the Pinyin (phonetic) representation of a sentence would help models "hear" the cloaked words. Surprisingly, the opposite occurred: adding Pinyin decreased performance across all models. The Pinyin acted as noise rather than a clarifying signal, suggesting that current LLMs lack the phonetic-semantic alignment necessary to recognize that different character strings can represent the same sound.

Discussion: Why Models Fail

Probability Chain Disruption

LLMs operate as next-token predictors based on statistical probabilities. Cloaking disrupts the expected "chain" of tokens. When a common insult like "too mean" (太贱) is replaced by the homophone for "peptide bond" (肽键), the model loses the semantic thread. It treats the sentence as a literal discussion of chemistry rather than a phonetic mask for an insult.

Linguistic and Cultural Context

Models performed better when instructions were provided in Chinese rather than English. This suggests that maintaining internal consistency between the "thought process" (the prompt) and the "task data" (the Chinese text) is crucial. Furthermore, case studies revealed that humans use deep cultural context to differentiate between offensive keywords and helpful intent (e.g., a user reporting domestic violence vs. a user being abusive), a nuance LLMs still struggle to capture.

Conclusion and Future Work

ToxiCloakCN demonstrates that current LLMs are highly vulnerable to phonetic and visual perturbations in Chinese offensive language. The study highlights several limitations, including a focus on Standard Mandarin and a static set of perturbations that do not account for character splitting or rapidly evolving slang.

Future research must move beyond literal token matching toward phonetic-aware embeddings and deeper semantic understanding. To effectively moderate the modern web, AI must learn to "hear" and "interpret" language with the same cultural intuition as the humans it seeks to protect.

Summary updated

Abstract

Detecting hate speech and offensive language is essential for maintaining a safe and respectful digital environment. This study examines the limitations of state-of-the-art large language models (LLMs) in identifying offensive content within systematically perturbed data, with a focus on Chinese, a language particularly susceptible to such perturbations. We introduce \textsf{ToxiCloakCN}, an enhanced dataset derived from ToxiCN, augmented with homophonic substitutions and emoji transformations, to test the robustness of LLMs against these cloaking perturbations. Our findings reveal that existing models significantly underperform in detecting offensive content when these perturbations are applied. We provide an in-depth analysis of how different types of offensive content are affected by these perturbations and explore the alignment between human and model explanations of offensiveness. Our work highlights the urgent need for more advanced techniques in offensive language detection to combat the evolving tactics used to evade detection mechanisms.
BibTeX
@inproceedings{xiao2024toxicloakcn,
  author = {Yunze Xiao and Yujia Hu and Kenny Tsu Wei Choo and Roy Ka-Wei Lee},
  title = {{ToxiCloakCN: Evaluating Robustness of Offensive Language Detection in Chinese with Cloaking Perturbations}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2024},
  doi = {10.18653/v1/2024.emnlp-main.345},
}

Loading related content...

Figure