TL;DR
This paper tackles the challenge of detecting adult content on Arabic Twitter, a task where existing methods, often developed for English, fall short. The authors introduce a novel, manually annotated dataset of Arabic Twitter accounts and demonstrate that even minimal user information (like usernames) can be highly effective for detection, with performance significantly boosted by incorporating a single tweet.
What Problem Does This Paper Address?
Social media platforms like Twitter have policies against adult content, yet spammers and malicious actors exploit these platforms to disseminate such material, particularly in regions where Arabic is widely used. Existing methods for detecting this content often rely on English-language lexicons or are generalized spam detection techniques that struggle with the nuances of Arabic dialects and the sophisticated evasion tactics employed by spammers. The sheer volume of user-generated content makes manual detection infeasible, necessitating automated solutions. This paper specifically addresses the need for effective, automated detection of adult content within the Arabic Twitter ecosystem.
Methods
The researchers employed a multi-pronged approach to tackle adult content detection:
- Dataset Creation: A key contribution is the creation of a new, manually annotated dataset. They collected Arabic tweets, filtered accounts based on country/city names in their profiles, and then had native Arabic speakers annotate accounts as "Adult" or "Non-Adult." This dataset comprises 6,000 adult accounts and 44,000 non-adult accounts, each with a sample tweet.
- Feature Engineering: They explored the utility of various information sources:
- User Information: Username, screen name, and user description.
- Tweet Content: A single random tweet from each account.
- Machine Learning Models: A range of models were evaluated:
- Traditional ML: Support Vector Machines (SVM) and Multinomial Naive Bayes (MNB).
- Deep Learning: FastText and contextual embedding models like Multilingual BERT and AraBERT (a BERT model specifically trained for Arabic).
- Analysis: Extensive analysis was performed on the dataset to understand the characteristics of adult content accounts, including word and emoji usage, tweet length, and geographical distribution.
Datasets
The primary dataset introduced in this paper is a collection of 6,000 manually annotated Arabic Twitter accounts that post adult content and 44,000 manually annotated non-adult accounts. Each account is paired with one random tweet. This dataset is made available for researchers.
Key Insights and Ideas
- Minimal Information is Powerful: Surprisingly, user information alone (specifically usernames) proved to be a strong indicator of adult content, achieving a high F1 score. This suggests that accounts aiming to attract users for adult content often embed clues in their public profiles.
- Arabic Nuances Matter: The paper highlights that methods developed for English are not directly transferable and that Arabic-specific approaches are crucial.
- User Info + Tweet is the Sweet Spot: While user information alone is effective, combining it with even a single tweet significantly boosts detection performance, indicating a synergistic effect.
- AraBERT Shines for Arabic: The AraBERT model, specifically trained for Arabic, demonstrated superior performance, underscoring the importance of language-specific models.
- Annotation Errors Exist: The error analysis revealed that a significant portion of misclassifications were due to initial annotation errors, suggesting that automated systems could potentially help refine human annotations over time.
Key Results
- User Information Alone: SVM models trained on usernames achieved a macro-averaged F1 score of 87.1%. Combining username and user description yielded an even higher 94.7% F1 score.
- Single Tweet: Using only a single tweet, SVM models achieved a macro-averaged F1 score of 88.9%.
- Combined Information: The best performance was achieved by combining user information (username + user description) with a single tweet, reaching a macro-averaged F1 score of 96.8%. The AraBERT model, using this combined information, was the top performer.
- Geographical Distribution: Saudi Arabia (SA) accounts for the largest proportion of both overall accounts and adult accounts in the dataset, suggesting a concentration of this type of content originating from the region.
Limitations and Future Work
- Limitations:
- The study focused solely on textual information, excluding multimedia (images) and social network features (followers/followees).
- The initial classification of vulgar tweets was based on a system that performed poorly on Arabic dialects, requiring manual re-annotation.
- Some complex cases where intuition is needed to connect profile information with tweets were difficult for classifiers to capture.
- Future Work:
- Investigate the inclusion of multimedia (images) and social network features for improved detection.
- Explore methods for identifying other types of undesirable content, such as unsolicited advertisements.
- Compare performance with models that depend solely on textual information.
Summary updated