Improving the sentiment classification of stock tweets

Sheng LiUniversity 2014

Abstract

This research focuses on improving stock tweet sentiment classification accuracy with the addition of the linguistic features of stock tweets. Stock prediction based on social media data has been popular in recent years, but none of the previous studies have provided a comprehensive understanding of the linguistic features of stock tweets. Hence, applying a simple statistical model to classifying the sentiment of stock tweets has reached a bottleneck. Thus, after analysing the linguistic features of stock tweets, this research used these features to train four machine learning classifiers. Each of them showed an improvement, and the best one achieved a 9.7% improvement compared to the baseline model. The main contributions of this research are fivefold: (a) it provides an in-depth linguistic analysis of stock tweets; (b) it gives a clear and comprehensive definition of stock tweets; (c) it provides a simple but effective way to automatically identify stock tweets; (d) it provides a simple but effective method of generating a localised sentiment keyword list; and (e) it demonstrates a significant improvement of stock tweet sentiment classification accuracy.

Introduction

Since the early 2010s, researchers have investigated the "collective mood" of Twitter as a predictor for stock market movements. However, early attempts encountered a performance bottleneck. Most studies relied on simple statistical models, such as "bag-of-words," which treat tweets as unordered collections of words. This approach ignores the unique, noisy, and highly informal linguistic structure of microblogging.

This research addresses three major gaps in the field:

  1. Definition: The lack of a rigorous academic definition for what constitutes a "stock tweet."
  2. Noise: The prevalence of spam and irrelevant chatter that pollutes sentiment data.
  3. Lexicon Mismatch: The failure of generic sentiment dictionaries (e.g., Harvard-IV) in financial contexts where terms like "short," "bullish," or "pullback" have domain-specific meanings.

Methodology

The study proposes a multi-layered approach to sentiment classification, moving beyond simple word counts to integrate deep linguistic features.

Hierarchy Classification Model

Instead of direct sentiment analysis, the research employs a three-level filter:

  1. Stock Level: Distinguishes if a tweet is about the stock market.
  2. Ticker Level: Identifies if the tweet pertains to a specific company (e.g., $GE).
  3. Polarity Level: Classifies the sentiment as positive, negative, or neutral.

Feature Extraction and Lexicon Generation

Linguistic features are categorized into Internal (word meaning and Part-of-Speech tags using the ARK tagger) and External (metadata such as word/character counts and the frequency of hashtags, cashtags, and retweets).

Furthermore, the study avoids pre-made dictionaries in favor of a Localized Sentiment Keyword List. Using Keyness Analysis (log-likelihood), the author identified words statistically unique to positive or negative stock tweets within the specific dataset, ensuring domain relevance.

Datasets and Experimental Setup

The study utilized a custom dataset collected via the Twitter Search API between April 2012 and February 2013, focusing on 30 companies from the Dow Jones Industrial Average (DJIA).

  • Training Set: 6,735 tweets related to General Electric ($GE) were manually annotated to create a "gold standard."
  • Validation Set: 702 tweets from Caterpillar Inc. ($CAT) were used to test the model's generalizability across different companies.

Four machine learning classifiers—Decision Trees, Random Forests, Naive Bayes, and Support Vector Machines (SVM)—were evaluated against a baseline bag-of-words model.

Results and Discussion

The integration of linguistic features yielded significant performance gains. The best-performing model, Random Forest, achieved an accuracy of 0.685, a 9.7% improvement over the baseline. Notably, at the "Stock Level" filter, classifiers reached near-perfect accuracy (0.999), effectively eliminating non-financial noise.

Key insights from the analysis include:

  • The Inadequacy of Cashtags: The presence of a cashtag (e.g., $AAPL) is not a definitive indicator of a stock tweet, as they are frequently used by spammers or for unrelated topics.
  • Linguistic Uniqueness: The most frequent words in stock tweets differ significantly from general English corpora, justifying the need for domain-specific lexicons.
  • Metadata Utility: External features like tweet length and cashtag frequency are strong indicators of legitimate financial discussion versus noise.
  • Refining Neutrality: By filtering out non-ticker-related tweets first, the "neutral" category becomes a more meaningful representation of objective financial information rather than unclassified noise.

Conclusion

This research demonstrates that accounting for the unique linguistic features of microblogging significantly enhances sentiment classification for financial markets. While the current study used a manually annotated dataset of 6,735 tweets, future work should explore scaling these methods through semi-supervised learning or real-time processing via the Twitter Streaming API. Untangling the specific contributions of linguistic features versus algorithmic choice remains an area for further investigation.

Summary updated

Abstract

This research focuses on improving stock tweet sentiment classification accuracy with the addition of the linguistic features of stock tweets. Stock prediction based on social media data has been popular in recent years, but none of the previous studies have provided a comprehensive understanding of the linguistic features of stock tweets. Hence, applying a simple statistical model to classifying the sentiment of stock tweets has reached a bottleneck. Thus, after analysing the linguistic features of stock tweets, this research used these features to train four machine learning classifiers. Each of them showed an improvement, and the best one achieved a 9.7% improvement compared to the baseline model. The main contributions of this research are fivefold: (a) it provides an in-­depth linguistic analysis of stock tweets; (b) it gives a clear and comprehensive definition of stock tweets; (c) it provides a simple but effective way to automatically identify stock tweets; (d) it provides a simple but effective method of generating a localised sentiment keyword list; and (e) it demonstrates a significant improvement of stock tweet sentiment classification accuracy.
BibTeX
@inproceedings{li2014improving,
  author = {Sheng Li},
  title = {{Improving the sentiment classification of stock tweets}},
  booktitle = {University},
  year = {2014},
}
Figure