JiraiBench: A Bilingual Benchmark for Evaluating Large Language Models' Detection of Human Self-Destructive Behavior Content in Jirai Community

Yunze Xiao, Tingyu He, Lionel Z. Wang, Yiming Ma, Xingyu Song, Xiaohang Xu, Irene Li, Ka Chung NgarXiv 2025 · DOI · arXiv

Abstract

This paper introduces JiraiBench, a novel cross-lingual benchmark designed to evaluate how well Large Language Models (LLMs) detect Risky Health Behaviors (RHB)—specifically Eating Disorders (ED), Non-Suicidal Self-Injury (NSSI), and Drug Misuse (DM)—within the transnational "Jirai" subculture. By analyzing content from Chinese and Japanese social media, the researchers explore the intersection of language, culture, and automated moderation. Their findings suggest that cultural framing in prompts is often more critical than linguistic matching for sensitive content detection.

Introduction

Detecting risky health behaviors online is a high-stakes challenge for public health and content moderation. However, existing research suffers from two major flaws: English-centricity and the use of "coded" language by vulnerable communities to evade standard filters.

The authors focus on the "Jirai" (landmine) subculture, a phenomenon spanning Chinese and Japanese social media. This community shares specific aesthetics and terminology (e.g., Jirai Kei) often intertwined with mental health struggles. Because the subculture is transnational but culturally specific, it serves as a perfect "stress test" for whether LLMs can understand the subtextual intent behind specialized slang.

Methodology

The researchers employed a multi-stage pipeline to build and test their benchmark, involving data collection, expert annotation, and model evaluation.

The multi-stage pipeline for JiraiBench data collection and annotation, including keyword search, expert labeling, and classification tasks.
  • Data Collection: Keyword-based searches were conducted on Sina Weibo (Chinese) and X/Twitter (Japanese) to identify posts related to DM, ED, and NSSI.
  • Expert Annotation: A team of six trained annotators, including community experts, labeled over 15,000 posts using a three-point ordinal scale (0: No tendency, 1: First-person expression, 2: Third-party description).
  • Model Evaluation: Four state-of-the-art LLMs—Llama-3.1 8B, Qwen-2.5 7B, DeepSeek-v3, and the fine-tuned JiraiLLM-Qwen—were tested under zero-shot and few-shot conditions.

The JiraiBench Dataset

JiraiBench consists of 10,419 Chinese posts from Sina Weibo and 5,000 Japanese posts from X. Interestingly, the Chinese dataset showed significantly higher rates of first-person Drug Misuse (30.55%) and Eating Disorder (15.04%) content compared to the Japanese set, highlighting regional differences in how these behaviors are expressed online.

Key Results and Insights

Comparative performance (Macro F1 scores) of various LLMs on Chinese and Japanese datasets across different prompt languages and shot configurations.

The "Japanese Prompt" Advantage

In a counterintuitive twist, Japanese instruction prompts often outperformed Chinese prompts when processing Chinese content. The authors hypothesize that because the Jirai subculture originated in Japan, Japanese prompts "activate" the correct cultural schema within the LLM more effectively than native-language prompts. This advantage persisted even when Kanji (shared characters) were removed, proving models respond to cultural context rather than simple orthographic matching.

Fine-tuning and Zero-Shot Performance

  • Fine-tuning Wins: The fine-tuned JiraiLLM-Qwen achieved the best results on Chinese data, with Macro F1 scores reaching 0.692 for Drug Misuse, significantly outperforming base models.
  • The Few-Shot Penalty: Surprisingly, providing examples (few-shot) often degraded performance compared to zero-shot. Specific examples might "interfere" with the model's attention, causing it to focus on narrow behavioral patterns rather than the diverse expressions found in the wild.
  • Safety-Refusal Barriers: GPT-4o was attempted as a baseline but consistently refused to classify the content due to internal safety filters. This highlights a hurdle for researchers: "closed" models are often too sanitized to study the very harms they are meant to prevent.

Conclusion

JiraiBench demonstrates that effective moderation of sensitive subcultures requires more than just linguistic translation; it requires cultural alignment. While fine-tuning offers significant gains, the study also reveals the potential of cross-lingual transfer, where models trained on one language's Jirai content can effectively identify similar patterns in another. Future work will aim to expand this framework to more languages and more nuanced annotation schemes to better support vulnerable online communities.

Summary updated

BibTeX
@misc{xiao2025jiraibench,
  author = {Yunze Xiao and Tingyu He and Lionel Z. Wang and Yiming Ma and Xingyu Song and Xiaohang Xu and Irene Li and Ka Chung Ng},
  title = {{JiraiBench: A Bilingual Benchmark for Evaluating Large Language Models' Detection of Human Self-Destructive Behavior Content in Jirai Community}},
  year = {2025},
  eprint = {2503.21679},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2503.21679},
}
Figure