TL;DR
This paper introduces VULGER, a novel lexicon of German vulgar and obscene language, built using a best-worst scaling approach to capture nuanced degrees of obscenity. This method overcomes the limitations of discrete categorical systems and provides a more robust foundation for NLP tasks like hate speech detection.
What Problem Does This Paper Address?
The proliferation of social media has led to an increase in uncivilized online discourse, including harassment, trolling, and cyberbullying. A critical aspect of this is the use of vulgar and obscene language, which is often socially banned from polite conversation. However, existing lexical resources for offensive language are primarily English-centric and often rely on simplistic categorical classifications (e.g., "offensive" vs. "not offensive"). This approach fails to capture the subtle but important differences in the degree of obscenity, which is crucial for accurate detection and mitigation of harmful online content. The paper aims to address this gap by creating a comprehensive, continuously scaled lexicon for German vulgar language.
Methods
The authors propose a three-step workflow for lexicon acquisition:
- Seed Lexicon Building: They start by compiling an initial set of German vulgar, rough, and pejorative terms from sources like Wiktionary and OpenThesaurus. This seed lexicon is then expanded by finding semantically similar words using word embeddings (FASTTEXT, UKP, TWITTER) derived from various corpora (Code Alltags, Dortmunder Chat-Korpus, Common Crawl, Wikipedia).
- Human Assessment (Best-Worst Scaling): This is the core novel contribution. Instead of asking annotators to assign discrete categories, they use Best-Worst Scaling (BWS). In BWS, annotators are presented with sets of items (n-tuples) and asked to identify the "best" (most neutral) and "worst" (most vulgar) item. This method, adapted from emotion scaling, allows for a more nuanced, continuous grading of obscenity. Crowdworkers from platforms like Figure Eight and Clickworker were employed for this task.
- Machine Learning for Automatic Extension: The scored seed lexicon is then used as training data for regression models (linear and ridge regression). These models, utilizing word embeddings as features, predict obscenity scores for new lexical items automatically extracted from larger corpora, including German swearword lists and hate speech corpora (IWG Hatespeech, GermEval 2018).
Datasets
- Seed Lexicon Sources: Wiktionary (German slice), OpenThesaurus (German).
- Corpora for Word Embeddings: Code Alltags (German email corpus), Dortmunder Chat-Korpus, Common Crawl, Wikipedia, Twitter.
- Corpora for Automatic Extension: German swearword lists (from hyperhero.com, insult.wiki, schimpfwoerter.de), IWG Hatespeech corpus (tweets annotated for hate speech), GermEval 2018 corpus (tweets annotated for offensive language).
- Newly Introduced Benchmark: The paper introduces VULGER, a lexicon of German vulgar language with continuous obscenity ratings. It comprises over 11,000 lexical items and 33,000 human ratings.
Key Insights and Ideas
- Continuous Scaling over Discrete Categories: The most significant insight is that obscenity is not a binary or multi-class categorical feature but rather a continuous spectrum. BWS effectively captures this nuance, providing richer data than traditional categorization.
- Leveraging Word Embeddings for Expansion: Word embeddings prove to be powerful tools for identifying semantically related words, enabling the expansion of the lexicon beyond manually curated lists.
- The Importance of "Lexical Noise": Including seemingly neutral words in the seed lexicon and during the annotation process helps to counterbalance the impact of overtly offensive terms, leading to more stable and reliable scaling.
- Challenges in German Lexicon Building: The paper highlights the scarcity of existing comprehensive lexical resources for German offensive language compared to English.
Key Results
- The VULGER lexicon was built, containing over 11,000 lexical items with approximately 33,000 human ratings.
- The Best-Worst Scaling approach yielded high inter-annotator reliability, with an average Pearson correlation of 0.9102 for split-half reliability.
- Ridge regression with FASTTEXT embeddings achieved the best performance in predicting obscenity scores, with an average Pearson correlation of 0.779.
- The automatic extension process, using the best model, added approximately 2,046 entries from swearword lists and 5,700 entries from hate speech corpora.
Limitations and Future Work
- Preliminary Characterization of Vulgar Language: The authors acknowledge that their characterization of vulgar language is preliminary and requires further in-depth clarification.
- Potential for "Lexical Noise" in Automatic Extension: While efforts were made to mitigate noise, the automatic extension process might still introduce some inaccuracies, especially with rare words or inflected forms.
- Limited Scope of Automatic Extension: The current automatic extension relies on word embeddings and might miss words for which no embeddings are available.
- Future Work:
- Experimental Validation: The authors plan to use and iteratively extend VULGER on text corpora with similar biases.
- Broader Discourse Context: They recognize that purely lexical approaches are insufficient and that broader discourse context is necessary for complete recognition of offensive language.
- Transfer Learning: Exploring methods like transfer learning to address sparse data problems.
- Human Validation of Automatically Extended Items: Feeding newly found lexical material back to annotators to compare human judgments with automatically predicted scores.
Summary updated