Abstract
Sentiment analysis is the computational study of reviews, emotions, and sentiments expressed in text. While deep neural networks have achieved significant results in this field, current methods primarily focus on the English language. Minority languages like Roman Urdu—Urdu written in the Latin alphabet—present unique challenges due to complex syntax and extreme lexical variations. This paper proposes a novel Self-attention Bidirectional LSTM (SA-BiLSTM) network to address these inconsistencies. By integrating Self-Attention to correlate sentence-wide context and BiLSTM to extract directional representations, the model tackles lexical noise effectively. Experimental results show that the SA-BiLSTM achieves high accuracy (up to 69.3%) with reduced computational overhead compared to state-of-the-art architectures.
Introduction: The "Wild West" of Roman Urdu
Sentiment analysis in Roman Urdu is complicated by a lack of standardized spelling. On social media and blogs, a single word like "beautiful" might appear as khubsurat, khubsoorat, or khoobsurat. This lexical variation creates significant noise for traditional machine learning models. Standard architectures like CNNs or unidirectional RNNs often fail to capture global context or become computationally inefficient when dealing with such diverse word representations. This research addresses these gaps by proposing a hybrid model designed for the specific linguistic nuances of Roman Urdu.
Methodology: The SA-BiLSTM Architecture
The proposed SA-BiLSTM architecture is designed to handle the sequential nature of text while focusing on the most sentiment-bearing tokens. The workflow consists of several integrated layers:
- Embedding & Positional Encoding: Since Roman Urdu lacks massive pre-trained models, 200-dimensional Word2Vec embeddings are used. Positional Encoding is added to maintain word order information, which is otherwise lost in attention mechanisms.
- Self-Attention Module: This layer correlates every word in a sentence with every other word, identifying which specific tokens (e.g., "acha" / good) most strongly influence the overall sentiment regardless of their position.
- Bidirectional LSTM (BiLSTM): The "attended" embeddings are processed in both forward and backward directions to capture sequential context and complex syntax.
- Classification: Global Average Pooling compresses the features before a sigmoid classifier produces the final binary sentiment (positive or negative).
Experimental Setup and Datasets
The researchers utilized a dataset of over 20,000 sentences scraped from Urdu blogs, social media, and news sites. To evaluate the impact of data quality, two specific versions were tested:
- Preprocessed Dataset: 10,000 sentences cleaned of non-textual symbols.
- Normalized Dataset: 3,000 sentences where lexical variations were manually unified (e.g., mapping all variations of "successful" to the single token kamyaab).
Results and Discussion
The SA-BiLSTM outperformed several baseline models, including Fasttext and standard LSTMs, across multiple metrics:
- Accuracy: The model achieved 68.4% on the preprocessed set and 69.3% on the normalized set.
- Efficiency: On the preprocessed dataset, SA-BiLSTM trained in 260 seconds, nearly twice as fast as a standard BiLSTM (465 seconds), demonstrating that the attention mechanism allows the model to "focus" more effectively with fewer resources.
- Data Quality vs. Quantity: A key finding was that the model performed better on the Normalized Dataset despite it being 73% smaller than the preprocessed one. This suggests that for non-standardized languages, linguistic normalization is more critical than raw data volume.
Conclusion
The SA-BiLSTM provides a robust framework for sentiment analysis in Roman Urdu by overcoming the limitations of unidirectional architectures and addressing lexical noise. While the lack of large-scale pre-trained embeddings remains a challenge, this hybrid approach sets a new benchmark for efficiency and accuracy in minority language processing. Future work will explore expanding the dataset scale and applying the architecture to more complex tasks like natural language inference.
Summary updated