Identifying News from Tweets

Jesse Freitas, Heng JiNLP+CSS @ EMNLP 2016 · OpenReview

TL;DR

This paper proposes a novel approach to identify newsworthy content within the massive, real-time stream of Twitter data. By focusing on journalism-based "news values" and employing active learning, the authors significantly improve the classification of trending topics as news and the ranking of individual tweets for newsworthiness, outperforming existing methods.

What Problem Does This Paper Address?

The sheer volume of tweets generated daily makes it challenging to sift through and identify genuinely newsworthy information. Previous attempts to automatically rank or classify tweets often relied on social network features (like user influence or retweet counts) or external web documents. However, these methods can be computationally expensive or fail to capture the inherent qualities of news content itself. This paper tackles the problem of identifying news from tweets by focusing on language-independent journalism and content-based features, aiming to provide a more robust and efficient solution.

Methods

The core of this paper lies in defining and leveraging "news values" derived from journalism theory (Galtung and Ruge, 1965) to identify newsworthy content. These values are translated into quantifiable features, including:

  • Negative Event Bias: Tweets about negative events are often considered more newsworthy.
  • Composition: Well-composed tweets are more likely to be news.
  • Elite Focus: References to elite nations, people, or organizations.
  • Human Interest: Content that affects or resonates with a broad audience.

These news values are encoded into features such as slang usage, sentiment, named entities (geo-political, people, companies), and parse tree characteristics. The authors propose a novel "newsworthiness classifier" trained using active learning. This approach iteratively selects the most informative new data points for training, improving model efficiency and performance. They apply this to two main tasks:

  1. Trend Classification: Identifying trending topics on Twitter that are newsworthy.
  2. Tweet Ranking: Ordering individual tweets within a topic based on their newsworthiness.

They utilize Support Vector Machines (SVMs) for both tasks, employing an eighth-order polynomial kernel and tuning parameters for optimal performance.

Datasets

The paper uses two primary datasets:

  1. Hurricane Irene Dataset: This dataset consists of 3,482 topically related tweets about Hurricane Irene, collected over three separate hours on different days during the storm. Tweets were annotated for newsworthiness on a scale of 1 to 3 by human annotators, with a focus on whether they contained new, noteworthy information at the time of tweeting.
  2. Trending Topics Dataset: This dataset comprises 2,593 trending topics collected over a month in 2012, with each topic containing up to 1,500 tweets. Topics are labeled as newsworthy or not newsworthy. This dataset is more diverse than the Hurricane Irene dataset, including tweets in multiple languages (translated to English).

Key Insights and Ideas

  • Journalism Theory is Applicable to Social Media: The paper demonstrates that established principles of journalism (news values) can be effectively applied to the informal and fast-paced environment of Twitter to identify newsworthy content.
  • Content Over Social Features: Focusing on the linguistic and journalistic qualities of the tweet content itself, rather than just social network signals (like retweets or follower counts), leads to better performance in identifying news.
  • Active Learning Boosts Efficiency: The use of active learning significantly improves the classification of trending topics, allowing for better performance with potentially less labeled data.
  • Negative Events and Elite Focus are Strong Indicators: Features related to negative events and references to prominent entities appear to be strong indicators of newsworthiness.

Key Results

  • Tweet Ranking: The proposed ordinal regression model, using journalism-based features, outperformed the traditional TextRank method and a heterogeneous network-based approach in terms of Normalized Discounted Cumulative Gain (nDCG).
  • Trend Classification: The "Content Features" model achieved better precision and F-score than a baseline approach, and the "Active Training" model further improved performance, reaching an F-score of 77.8%. This highlights the effectiveness of active learning in this context.

Limitations and Future Work

  • Feature Overlap: The authors note that features used for trend classification and tweet ranking were highly correlated, suggesting potential redundancy or shared underlying signals.
  • Zubiaga's Features for Ranking: The features used by Zubiaga for trending topics did not translate well to single-tweet ranking, limiting the evaluation of query-by-committee methods on the ordinal regression task.
  • Future Work: The authors express interest in exploring query-by-committee methods for ordinal regression and integrating web documents with their content-based approach to further enhance news summarization tasks. They also suggest that while their features are not "advanced," their application based on journalism standards offers a valuable perspective compared to purely social-feature-driven methods.

Summary updated

BibTeX
@inproceedings{freitas2016identifying,
  author = {Jesse Freitas and Heng Ji},
  title = {{Identifying News from Tweets}},
  booktitle = {NLP+CSS @ EMNLP},
  year = {2016},
  url = {https://openreview.net/forum?id=NCiXJx5Q460},
}
Figure