TL;DR
This paper addresses the critical lack of resources for detecting offensive language and hate speech in Spanish, particularly outside of Spain. The authors introduce a new, manually annotated Chilean Twitter dataset and a comparative framework to facilitate research and model development in this under-resourced area.
What Problem Does This Paper Address?
The field of offensive language and hate speech detection is heavily skewed towards English, with a significant scarcity of resources for other languages, especially Spanish. Existing Spanish datasets are limited in geographical scope and often fail to capture the diverse socio-cultural nuances that influence hate speech. This lack of diverse, representative data hinders the development of robust cross-lingual and cross-cultural detection models, making it difficult to effectively moderate online content globally. The paper aims to bridge this gap by providing new data and tools to enable more comprehensive research.
Methods
The core of the paper's contribution lies in its creation of a new dataset and a framework for comparative analysis.
- Dataset Creation: The authors manually annotated 9834 tweets from Chile, focusing on offensive language and hate speech. This dataset is unique in its origin from South America and aims to capture specific regional vocabulary and targets of hate. The annotation process involved three native Chilean annotators who underwent rigorous training to ensure high inter-annotator agreement (over 90% agreement and Krippendorff's alpha > 0.7).
- Comparative Framework: To facilitate research and model comparison, the authors developed:
- A Python library that standardizes the interfaces of five published cross-lingual hate speech detection models, making them easier to use and compare.
- A Codalab competition to provide a standardized environment for researchers to benchmark new models against existing ones.
- Resource Repository: They also organized existing datasets into a structured repository, making it easier for researchers to find and access relevant resources.
Datasets
- Newly Introduced Dataset:
- Chilean Dataset: 9834 manually annotated tweets from Chile, labeled for various categories of offensive content, including hate speech, insults, and profanity. It also includes contextual information like the target of offensive speech and the use of irony.
- Datasets Used for Evaluation:
- English Dataset: Not explicitly detailed, but mentioned as part of the literature.
- Spanish Dataset (ES): The SemEval 2019 Spanish dataset (Basile et al., 2019), with examples originating from Spain.
- Chilean Dataset (CL): The newly created dataset.
Key Insights and Ideas
- Socio-cultural Context is Crucial: Hate speech is deeply rooted in socio-cultural contexts, meaning models trained on data from one region may not generalize well to others, even within the same language. This highlights the need for geographically diverse datasets.
- Spanish Language Diversity is a Major Hurdle: The paper emphasizes that variations in Spanish across different countries make models poorly generalizable when trained on limited regional data.
- Lack of Centralized Resources: The authors point out the difficulty researchers face in finding and comparing existing datasets and models due to the lack of centralized repositories.
- Cross-lingual Transfer is Challenging: Even with models designed for cross-lingual transfer, performance can vary significantly depending on the source and target datasets, suggesting that simply translating data isn't always sufficient.
Key Results
- Chilean Dataset Performance: The paper demonstrates that models trained on general Spanish (ES) datasets performed better than those trained on the Chilean (CL) dataset when tested on the CL dataset. This suggests that the Chilean variant and its specific hate speech targets are not well-represented in existing general Spanish datasets.
- Cross-cultural Performance: Experiments showed that knowledge transfer between the Spanish (ES) and Chilean (CL) datasets was generally poor, with results lower than within-dataset experiments. This reinforces the idea of significant differences between Spanish variants.
- Model Performance: The ECML20 model, which uses LASER representations, showed the overall best results in cross-lingual experiments, even with a simple structure.
- Baseline Performance: The paper evaluated several baselines, including dummy classifiers, seed classifiers, and an EMB+RF approach using Spanish FastText embeddings, finding the latter to perform best among the baselines.
Limitations and Future Work
- Gender Inference Caution: The annotators inferred gender from names and pronouns, acknowledging that this should be used carefully to avoid unfair models due to the non-binary nature of gender.
- Limited Spanish Representation: The authors note that their repository, while comprehensive, still shows a low representation of different types of Spanish spoken worldwide, indicating a need for more datasets from various Spanish-speaking regions.
- Future Work: The paper sets up a Codalab competition to encourage further research and comparison of new models. The developed library and repository are intended to facilitate future work in multilingual hate speech detection.
Summary updated