Introduction
In June 2020, Antonio Radić, known as Agadmator, the world’s most popular chess YouTuber, was temporarily banned for "harmful and dangerous" content. This incident highlighted a critical flaw in modern Natural Language Processing (NLP): the inability of automated hate speech detectors to handle domain-specific language and color polysemy. In the context of chess, terms like "black," "white," and "attack" are technical descriptors of game pieces and strategies, yet off-the-shelf classifiers frequently misinterpret them as racial vitriol. This paper investigates this phenomenon, quantifying the extreme false positive rates of existing models and introducing a new adversarial dataset to improve model robustness.
Methodology
The researchers employed a multi-stage approach to analyze domain sensitivity in hate speech detection. First, a large-scale corpus of chess-related YouTube comments was collected. Second, two distinct classifiers were evaluated: $M_{twitter}$, a standard model trained on general social media data, and $M_{stormfront}$, a BERT-based model trained on white supremacist forum data. Finally, a subset of flagged comments underwent human annotation to establish ground truth. Additionally, the researchers utilized Skip-gram word embeddings to analyze latent biases through analogy testing (e.g., comparing associations in chess vs. news datasets).
Data Collection and Corpus Description
The researchers compiled $D_{chess}$, a corpus consisting of 681,995 comments from 8,818 videos. These videos were sourced from five influential chess channels, providing a diverse range of technical and casual commentary.
In addition to the raw corpus, the authors released an "Adversarial Set" of 1,000 human-annotated comments. These comments were specifically chosen because they were flagged as hate speech by the baseline models but were confirmed to be benign chess discussions. This dataset serves as a benchmark for testing the domain sensitivity of future NLP models.
Results and Analysis
The evaluation revealed a staggering discrepancy between automated classification and human judgment. The models exhibited a false positive rate (FPR) of approximately 82.4%. While $M_{twitter}$ flagged 1.25% of the corpus as positive, human evaluation showed that only 5% of those flags were true positives.
Interestingly, $M_{stormfront}$ proved more conservative, flagging only 0.43% of the data. The researchers hypothesize that exposure to a wider variety of actual hate speech allows the model to better distinguish between genuine vitriol and technical jargon.
A key finding emerged from the embedding analysis. While news datasets often reflect societal biases (e.g., associating "black" with "slave"), the chess embeddings demonstrated a "rare equality." In chess, both "black" and "white" are equally subject to "captures," "threats," and "kills," leading the model to predict black:slave :: white:slave. This suggests that the structured nature of the game creates a linguistic environment where color-based terms are treated with a level of parity rarely seen in general social datasets.
Discussion and Future Work
The high false positive rates observed in this study highlight the critical need for domain-aware NLP systems. Current models lack the context to recognize that "White’s attack on Black" is a tactical observation rather than a threat. Future research should focus on developing classifiers that can identify the semantic domain before applying filters. Furthermore, the "rare equality" found in chess embeddings suggests that certain structured domains could serve as templates for debiasing general-purpose models. Expanding this research to other domains with linguistic overlaps, such as martial arts or military history, remains a priority for future work.
Summary updated