TL;DR
This paper tackles the challenge of classifying sexist tweets, a task often hindered by scarce and imbalanced datasets. The authors propose a novel approach that leverages knowledge graphs (ConceptNet and Wikidata) to augment existing data and generate new, synthetic tweets, significantly improving classification performance across various machine learning models.
What Problem Does This Paper Address?
Classifying nuanced and often subtle forms of harmful language, like sexism in online text, is a critical but challenging task in natural language processing (NLP). A major hurdle is the scarcity and imbalance of labeled datasets. Machine learning models, especially deep learning ones, typically require large amounts of high-quality data to perform well. When datasets are small or heavily skewed towards certain classes, models tend to underperform, leading to misclassifications. This paper specifically addresses the problem of classifying sexist tweets, where obtaining a comprehensive and balanced dataset is difficult. The authors aim to overcome this data limitation by enriching the existing data and generating synthetic examples that capture the essence of sexist language.
Methods
The core of this paper lies in its innovative use of knowledge graphs for text augmentation and generation. They employ two primary strategies:
- Text Generation: New tweets are created by replacing words in existing tweets with related concepts found in ConceptNet. This is done using specific ConceptNet relations (e.g., "IsA," "RelatedTo," "HasProperty") to ensure semantic coherence. Three variations of this method are explored:
- All Words Replacement (AWR): Replaces all non-stop words with their related concepts.
- Verb Replacement (VR): Replaces only verbs with their synonyms.
- Noun Replacement (NR): Replaces only nouns with their related concepts.
- Text Augmentation: Instead of generating entirely new tweets, this method enriches existing ones.
- ConceptNet Augmentation: Adds concepts from ConceptNet (specifically, the top 10 related concepts based on weight for the "IsA" relation) to the end of each tweet. This increases the length of the tweets without changing the number of samples.
- ConceptNet + Wikidata Augmentation: This more sophisticated approach combines ConceptNet with Wikidata. It involves adding related concepts from ConceptNet and then using Wikidata's concept descriptions. To bridge the gap between ConceptNet and Wikidata (which lack direct mapping), they use cosine similarity of word vectors to find the most relevant Wikidata descriptions for the tweet's concepts.
Datasets
The study utilizes a dataset of sexist tweets previously collected and categorized by Sharifirad and Matwin (2018). This dataset was divided into four categories: indirect harassment, information threat, sexual harassment, and physical harassment. For evaluation, the authors focused on three of these categories after noting that the "information threat" category had too few samples to be reliably used.
Key Insights and Ideas
- Knowledge Graphs as Data Augmentors: The paper convincingly demonstrates that knowledge graphs like ConceptNet and Wikidata are powerful tools for overcoming data scarcity in NLP tasks, particularly for nuanced language classification.
- Text Generation Outperforms Augmentation: The results suggest that generating entirely new, semantically rich tweets through word replacement (especially AWR) leads to greater performance improvements than simply augmenting existing tweets with additional concepts or descriptions.
- ConceptNet's Richness: ConceptNet, with its broad coverage of common-sense knowledge and relationships, proves to be a valuable resource for generating diverse and relevant tweet variations.
- Wikidata's Descriptive Power: Integrating Wikidata's detailed concept descriptions, even with the added complexity of mapping, further enhances the augmentation process, though not to the same extent as full text generation.
- Generalizability: The proposed methods are not limited to sexist tweet classification and are presented as readily applicable to other text classification problems, especially those dealing with scarce or imbalanced data, such as hate speech or abusive language.
Key Results
The paper reports significant improvements in sexist tweet classification accuracy across various machine learning models (OVR, SVM, Naive Bayes, LSTM, CNN) after applying their text generation and augmentation techniques.
- Text Generation: The "All Words Replacement" (AWR) method, using ConceptNet, achieved the highest performance boost. For instance, with LSTM and CNN classifiers, AWR achieved accuracies of 0.98, a substantial increase from the original dataset's best accuracy of 0.75 (achieved by CNN). Verb Replacement (VR) and Noun Replacement (NR) also showed considerable improvements over the baseline.
- Text Augmentation: While not as impactful as text generation, augmentation methods also showed improvements. Augmenting with ConceptNet alone yielded accuracies around 0.90 (with LSTM/CNN), while combining ConceptNet and Wikidata reached 0.93 (with LSTM).
Limitations and Future Work
- Limitations:
- The paper acknowledges that ConceptNet and Wikidata are not perfectly consistent or complete.
- The "information threat" category in their dataset was too small to be effectively used for classification.
- The text generation methods might produce "meaningless tweets" if relationships other than specific ones (like "FromOf" and "IsA") are used.
- Future Work:
- Adding more words and descriptions from Wikidata for text augmentation.
- Applying these methods to other abusive and hate speech datasets.
- Combining ConceptNet, Wikidata, and emoji ontologies for richer augmentation and generation.
- Investigating the impact of these methods on larger datasets.
Summary updated