TL;DR
This paper presents a machine learning system for detecting homophobia and transphobia in social media comments, specifically from YouTube. The authors leverage TF-IDF for text representation and Support Vector Machines (SVM) as their primary classifier, achieving strong performance, particularly on Tamil and mixed-language datasets.
What Problem Does This Paper Address?
The internet, while a vast source of information, also serves as a breeding ground for misinformation and harmful content. Identifying and flagging hate speech, including homophobia and transphobia, is a critical challenge, especially on platforms like YouTube where users can freely share opinions. This paper tackles the problem of automatically detecting such harmful content within social media comments, aiming to foster a more inclusive online environment. The authors highlight the difficulty of this task, particularly in low-resource languages and code-mixed text, where users might switch between languages or use English script for their native language.
Methods
The proposed system employs a standard machine learning pipeline.
- Text Cleaning: Basic preprocessing steps are applied to clean the raw text, including removing hashtags, special symbols, URLs, whitespace, and repeated characters.
- Feature Engineering: The Term Frequency-Inverse Document Frequency (TF-IDF) technique is used to convert text comments into numerical vectors. This method captures the importance of words within a document relative to their frequency across a corpus. The authors specifically mention using TF-IDF with bigram models for feature extraction.
- Classification: The core of the system relies on machine learning classifiers. The paper explores several algorithms, including:
- Support Vector Machine (SVM): This is the primary classifier used, with a focus on its ability to handle wide feature sets and binary classification problems. The authors specifically tested a linear kernel with a regularization parameter of 5.
- Random Forest (RF): An ensemble method based on decision trees.
- Passive Aggressive Classifier (PA): An online learning algorithm suitable for scenarios requiring continuous data updates.
- Gaussian Naïve Bayes (GNB): A probabilistic classifier.
- Multi-Layer Perceptron (MLP): A feed-forward neural network with a hidden layer of 20 nodes and a logistic activation function.
The authors also mention exploring an ensemble approach using hard voting, though the detailed results focus on individual classifiers.
Datasets
The dataset used for this task was provided for the shared task (Chakravarthi et al., 2022) and consists of YouTube comments. It is divided into three categories:
- English: Comments written solely in English.
- Tamil: Comments written solely in Tamil.
- Tamil-English: Code-mixed comments, where users switch between Tamil and English.
The dataset was scraped between August 2020 and February 2021 and contains approximately 22,000 comments in total. The extracted comments include both homophobic and non-anti-LGBT+ text.
Key Insights and Ideas
- SVM's Robustness: The Support Vector Machine (SVM) consistently outperformed other classifiers across all language datasets, demonstrating its effectiveness for this specific task.
- Tamil Language Performance: The Tamil dataset showed the best performance for the proposed model, suggesting that the chosen features and classifier are particularly well-suited for this language.
- Code-Mixing Challenge: While the model achieved a respectable 8th rank on the mixed-language Tamil-English dataset, it highlights the inherent complexity of processing code-mixed text for hate speech detection.
- TF-IDF Effectiveness: The use of TF-IDF with bigrams proved to be a strong feature representation method for capturing relevant linguistic patterns in social media comments.
Key Results
The paper reports F1-scores as the primary evaluation metric.
- Development Phase: In the development phase, SVM achieved the highest F1-scores for Tamil (0.85) and Tamil-English (0.51) datasets, while performing moderately on English (0.43).
- Final Results (SVM): For the final test set:
- English: Accuracy of 0.94, Weighted F1-score of 0.91, Rank 11.
- Tamil: Accuracy of 0.92, Weighted F1-score of 0.92, Rank 2.
- Tamil-English: Accuracy of 0.90, Weighted F1-score of 0.88, Rank 8.
The weighted F1-score generally showed high values, indicating good overall performance, especially for the Tamil dataset.
Limitations and Future Work
- Limitations:
- The paper acknowledges that the English dataset yielded the worst performance, suggesting potential areas for improvement.
- While the model performed well on Tamil, its rank of 2nd indicates there's still room for enhancement.
- Future Work: The authors suggest that applying more complex systems, such as deep learning models with different architectures, could further improve performance. They also propose exploring other word representation models like word embeddings as an alternative to TF-IDF for potentially richer feature extraction.
Summary updated