Topic and Sentiment Analysis on OSNs: a Case Study of Advertising Strategies on Twitter

Shana Dacres, Hamed Haddadi, Matthew PurverarXiv 2013 · arXiv

Abstract

This study evaluates the effectiveness of "promoted tweets" versus "promoted trends" on Twitter, focusing on user engagement and sentiment. By employing machine learning for topic classification and sentiment analysis, the researchers demonstrate that while promoted trends maximize volume, promoted tweets drive higher quality engagement and positive sentiment. The findings underscore the necessity of robust filtering to overcome the inherent noise in social media data.

Introduction

As brands increasingly leverage Online Social Networks (OSNs) for advertising, measuring the true success of these campaigns remains a significant challenge. Traditional analytics often rely on simplistic metrics like retweet counts or basic keyword matching, which can be highly misleading. This paper addresses two core challenges: topic ambiguity (e.g., the keyword "O2" referring to a brand, a chemical element, or a venue) and sentiment complexity (the difficulty of capturing nuance and slang in short-form text).

Methodology

The researchers developed a multi-stage pipeline to process approximately 180,000 tweets across 11 diverse brands. The methodology focused on two primary areas:

Topic Classification

The study compared simple keyword filtering against supervised machine learning classifiers, including Naive Bayes (NB), J48, and ID3. A Naive Bayes classifier, trained on manually labeled data, was found to significantly outperform keyword-based methods.

Sentiment Analysis

Three approaches were benchmarked: lexicon-based tools (Data Science Toolkit), hybrid models (SentiStrength), and ML-based models (Chatterbox’s Sentimental API). The ML-based approach proved superior due to its ability to interpret sentiment within hashtags and informal language.

Results and Insights

The analysis revealed critical differences between advertising formats and the necessity of advanced data filtering.

The Impact of Filtering

The "O2" case study highlighted the severity of data noise: over 70% of tweets containing the keyword were irrelevant to the brand. The Naive Bayes model achieved 84% accuracy in identifying relevant tweets, whereas the ML-based sentiment tool also reached 84% accuracy, far exceeding the <50% accuracy of lexicon-based baselines.

Figure 4: Comparison of accuracy, recall, and precision for filtered vs. non-filtered data.
  • Volume vs. Sentiment: Promoted trends generate high volume ("loudness") but often attract neutral or objective content. In contrast, promoted tweets result in lower volume but significantly higher positive sentiment (averaging 58% on the first day) and more direct brand mentions (@brand).
  • Engagement Patterns: Both strategies see a peak on the first day, followed by a decline. However, promoted tweets exhibit a more "bouncy" or sustained engagement pattern over time compared to the steady decline of trends.

Conclusion

The study concludes that while promoted trends are effective for broad visibility, promoted tweets are superior for driving positive sentiment and direct brand interaction. Furthermore, the research emphasizes that without robust ML-based filtering, brands are likely overestimating their social media engagement by a massive margin due to irrelevant "noise." Future work will look to integrate social graph influence and higher temporal granularity to further refine these insights.

Summary updated

BibTeX
@misc{dacres2013topic,
  author = {Shana Dacres and Hamed Haddadi and Matthew Purver},
  title = {{Topic and Sentiment Analysis on OSNs: a Case Study of Advertising Strategies on Twitter}},
  year = {2013},
  eprint = {1312.6635},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1312.6635},
}
Figure