Classification of Tweets about Reported Events using Neural Networks

Kiminobu Makino, Yuka Takei, Taro Miyazaki, Jun GotoEMNLP 2018

TL;DR

This paper tackles the challenge of automatically classifying tweets into "Reported-event" (citing news sources) and "New-information" (original reports) categories to aid TV news production. They propose a novel neural network architecture that combines character and word sequences, achieving an impressive 88% F1 score.

What Problem Does This Paper Address?

News organizations increasingly rely on social media, particularly Twitter, to gather information for their reports. However, the sheer volume of tweets makes it difficult to sift through and identify relevant content. This paper focuses on a specific need within news production: distinguishing between tweets that report on an event by citing established news agencies or user-generated content sites (REPORTED) and those that provide new information or opinions about an event (NEW). Program creators have different needs for these two types of tweets; for instance, those seeking initial reports might only want NEW tweets, while others might want to track public reaction, which would be found in NEW tweets. Existing systems can extract "event-describing tweets" (EVENT), but classifying them into these two crucial categories for news production is a novel task.

Methods

The authors propose a two-stage system. First, an existing extraction process (Miyazaki et al., 2017) is used to filter out non-event tweets and identify EVENT tweets. The core contribution lies in the second stage: a novel neural network architecture for classifying these EVENT tweets into REPORTED and NEW.

The proposed method, termed "Concat Input NN," uniquely combines character and word sequences from tweets. This approach aims to leverage the strengths of both:

  • Character sequences: Handle noisy text and languages without clear word boundaries (like Japanese) without needing complex morphological analyzers. They also have fewer "unknown" characters than words.
  • Word sequences: Capture richer semantic meaning, as words often carry more meaning than individual characters.

The architecture processes character and word sequences independently through sequential neural networks (like LSTM or CNN) and then concatenates their outputs before feeding them into a final feed-forward network for classification. This contrasts with existing methods that use either character or word sequences exclusively, or combine them in less integrated ways.

Datasets

The paper utilizes a large dataset of randomly sampled Japanese tweets.

  • Extraction Process Training: 19,962 manually extracted "event-describing tweets" (positive samples) and 1,524,155 randomly sampled tweets (negative samples).
  • Classification Training & Testing: 44,670 tweets for training and 10,000 tweets for testing, all extracted as EVENT tweets. These tweets were manually annotated into three categories: REPORTED with explicit sources (I-1), REPORTED without explicit sources (I-2), and NEW (II). For the classification task, I-1 and I-2 were combined into a single REPORTED category.

Key Insights and Ideas

  • The Value of Hybrid Input: Combining character and word sequences in a neural network significantly improves classification performance compared to using either alone. This is particularly beneficial for noisy, user-generated text like tweets.
  • Character Sequences for Robustness: For languages like Japanese, where word segmentation can be challenging and tweets are often informal, character-level processing offers a robust alternative to word-level processing, avoiding the need for complex linguistic tools.
  • Task-Specific Classification is Crucial: The distinction between "Reported-event" and "New-information" tweets is not just an academic exercise; it directly addresses a practical need in news production, enabling more efficient information acquisition for TV programs.
  • CNNs as a Speed-Accuracy Trade-off: While LSTMs generally achieve slightly higher recall, CNNs, especially when combined with the proposed input method, offer a better balance of speed and accuracy, making them more suitable for real-time news production systems.

Key Results

  • The proposed "Concat Input NN" method achieved an F1 score of 88%, outperforming conventional character-only and word-only input methods across various NN architectures (FFNN, LSTM, CNN).
  • The CNN architecture with the proposed concat input method achieved a good balance of speed and accuracy, with a training time of approximately 37 minutes for 5 epochs.
  • The proposed method significantly reduced the recall difference between REPORTED tweets with and without explicit sources, indicating its effectiveness in handling variations in tweet structure.
  • Compared to baseline methods like keyword filtering and SVMs, the proposed NN architectures generally showed superior performance, especially with sufficient training data.

Limitations and Future Work

  • False Negatives: The system still incorrectly extracted some tweets, including opinions about news or reports that were not cited.
  • Future Work: The authors plan to extend the system to handle tweets mentioning the same topic, which is relevant for event detection tasks. They also aim to increase dataset reliability by using more annotators and explore other tasks for evaluating their proposed methods. The paper also notes that for real-world use, training data can be gathered based on feedback from TV program production, suggesting an iterative improvement process.

Summary updated

BibTeX
@inproceedings{makino2018classification,
  author = {Kiminobu Makino and Yuka Takei and Taro Miyazaki and Jun Goto},
  title = {{Classification of Tweets about Reported Events using Neural Networks}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2018},
}
Figure