Introduction
Hate speech detection is a critical task in natural language processing, yet existing resources often fall short in three key areas: language diversity, the nuance of implicit hate, and the inherent bias of human annotators. Most datasets are English-centric, overlook subtle forms of prejudice like sarcasm or stereotypes, and fail to account for the cognitive biases of the people labeling the data.
To address these gaps, researchers have introduced K-HATERS, the largest Korean hate speech dataset to date. It comprises 192,158 news comments annotated with fine-grained, target-specific ratings. Crucially, the study also investigates how annotators' cognitive profiles—measured via the Cognitive Reflection Test (CRT)—influence the quality and fairness of the resulting AI models.
Methodology: Beyond Binary Labels
The K-HATERS dataset moves beyond simple "hate/not hate" classifications by employing a multi-layered annotation strategy designed to capture the complexity of online discourse.
Annotation Pipeline
Annotators evaluated comments using a three-point Likert scale (0: None, 1: Weakly/Implicitly offensive, 2: Obviously/Seriously offensive). They also identified 11 specific variables, including the target of the hate (e.g., gender, age, politics) and the type of offense (e.g., insult, obscenity).
Label Transformation and Modeling
To facilitate standard classification tasks, raw ratings were transformed into four distinct categories: Normal, Offensive, Level-1 Hate (implicit), and Level-2 Hate (explicit). The researchers fine-tuned Korean-specific BERT models (KcBERT, KPFBERT, and KLUE-BERT). The most advanced variant, the H+T+R model, utilizes multi-task learning to simultaneously predict the hate category, the target group, and the specific text spans (rationales) that trigger the offensive label.
The K-HATERS Dataset
Sourced from Naver News sections covering politics, society, and world events, K-HATERS is 1.75 times larger than previous Korean corpora. It provides a robust benchmark alongside existing datasets like BEEP!, KoLD, and K-MHaS. Analysis shows that hate speech in "hard news" (politics) is typically group-targeted, whereas "soft news" (entertainment) tends to target individuals.
Key Results and Fairness
The H+T (Hate + Target) model emerged as the top performer, achieving a Micro F1 of 0.681 and a Macro F1 of 0.611. A significant finding was that models trained on transformed labels significantly outperformed those using raw variables, particularly in terms of fairness and plausibility.
The Role of the "Reflective" Annotator
A unique contribution of this work is the use of the Cognitive Reflection Test (CRT) as a proxy for annotator quality. The CRT measures an individual's ability to suppress intuitive, often biased, responses in favor of reflective thinking. The study found that models trained on data from "low-CRT" annotators—those more prone to impulsive judgment—were less accurate and exhibited higher predictive bias. This suggests that selecting reflective annotators is as important as the dataset's scale.
Conclusion and Future Directions
K-HATERS provides a massive, high-quality resource for Korean NLP. While currently limited to Naver News, it sets a new standard for dataset creation by incorporating cognitive psychology into the annotation process. Future work aims to integrate CRT scores directly into model training to automatically mitigate the impact of biased labels.
Summary updated