Abstract
Sentiment analysis on social media is frequently hampered by the prevalence of regional slang, abbreviations, and linguistic ambiguity. This paper introduces the Proposed Textual Preprocessing Method (PTPM), a specialized framework designed to improve sentiment detection in Nigerian tweets ("Naija-tweets"). By leveraging an Integrated Knowledge Base (IKB) that includes localized sources like Naijalingo and employing an adapted Lesk algorithm for word sense disambiguation, the PTPM framework effectively resolves the meanings of noisy terms. Experimental results demonstrate that PTPM significantly outperforms existing methods, achieving a peak accuracy of 99.17% when paired with Convolutional Neural Networks (CNN).
Introduction
Automated sentiment analysis—categorizing text as positive, negative, or neutral—faces significant hurdles in the context of social media. Tweets are characterized by brevity and "noisy" content, including slang, acronyms, and emoticons. These challenges are amplified in regional contexts like Nigeria, where standard Natural Language Processing (NLP) tools often fail to recognize local dialects or "adulterated" English. Furthermore, many slang terms are ambiguous; a single abbreviation can have multiple meanings depending on the context. Traditional preprocessing often relies on simple string replacement, which fails to account for this polysemy.
Methodology: The PTPM Framework
To address these gaps, we developed the Proposed Textual Preprocessing Method (PTPM). This framework enhances the standard NLP pipeline with two critical components: an Integrated Knowledge Base and a context-aware disambiguation algorithm.
Integrated Knowledge Base (IKB)
The IKB serves as a centralized API aggregating data from three primary sources: Naijalingo (providing essential Nigerian context), Urban Dictionary, and Internetslang.com. This comprehensive repository allows the system to recognize over two million slang terms and emoticons, ensuring broad coverage of both global and regional internet vernacular.
Adapted Lesk Algorithm
The core innovation of PTPM is the use of an adapted Lesk algorithm to handle linguistic ambiguity. When a slang term with multiple definitions is identified in the IKB, the system evaluates the surrounding context of the tweet. By calculating a "relatedness" score based on word intersections between the tweet and the IKB's usage examples, the algorithm selects the most appropriate definition for that specific instance.
Experimental Setup
The framework was evaluated using a curated dataset of 10,000 Nigerian tweets related to political keywords (e.g., "policy," "government"). These tweets were manually labeled by three experts into positive and negative categories. The dataset was split into training (80%), testing (10%), and development (10%) sets. We compared PTPM against the General Textual Preprocessing Method (GTPM) and the Arc2Vec framework across several classifiers: Support Vector Machines (SVM), Multi-Layer Perceptrons (MLP), and Convolutional Neural Networks (CNN).
Results and Discussion
The PTPM framework consistently yielded superior performance across all tested classifiers and feature sets (unigrams, bigrams, and combinations).
Performance Metrics
The most significant gains were observed with deep learning architectures. The combination of PTPM and CNN achieved a remarkable accuracy of 99.17%, outperforming both GTPM and Arc2Vec benchmarks.
Key Insights
- Localization is Essential: The inclusion of Naijalingo was pivotal. General slang dictionaries often overlook the nuances of West African social media, leading to misclassification.
- Disambiguation Reduces Noise: Treating slang as a Word Sense Disambiguation (WSD) problem via the Lesk algorithm drastically reduces data noise compared to simple replacement methods.
- Deep Learning Synergy: While PTPM improved all models, its synergy with CNNs suggests that cleaner, semantically enriched input allows deep learning models to extract more effective features.
Conclusion
The PTPM framework demonstrates that resolving regional slang ambiguity is a vital step for accurate sentiment analysis in localized contexts. Future work will focus on the computational scalability of the IKB and testing the framework's generalizability to other regional dialects, such as South African or Kenyan English, by integrating corresponding local knowledge bases.
Summary updated