Analysis of Irony and Stereotype Spreaders Based On Convolutional Neural Networks

Dong Yuan, Wenyin Yang, Li Ma, Zhenlin Tan, Qidi LaoCLEF (Working Notes) 2022 · OpenReview

TL;DR

This paper tackles the challenge of automatically identifying Twitter users who spread irony and stereotypes. The authors propose a two-layer Convolutional Neural Network (CNN) model that achieves good accuracy by effectively extracting semantic features from text. This work contributes to the ongoing effort to understand and mitigate harmful online discourse.

What Problem Does This Paper Address?

The internet, while a powerful tool for communication, is also a breeding ground for various forms of speech, including irony and stereotypes. Identifying users who actively spread these types of content is crucial for understanding online dynamics and potentially mitigating their negative impact. The PAN 2022 competition specifically focused on the task of "Profiling Irony and Stereotype Spreaders on Twitter," aiming to develop models that can automatically classify users based on their tweets. This paper addresses this challenge by proposing a deep learning approach to automatically identify such spreaders.

Methods

The proposed model employs a two-stage approach: text preprocessing and feature extraction.

Text Preprocessing: * XML to Text Conversion: The raw XML data from the dataset is converted into plain text files, organized into folders based on the "I" (Irony/Stereotype Spreader) or "NI" (Not Irony/Stereotype Spreader) labels. * Text Vectorization: The text is then processed using TensorFlow's Text Vectorization layer. This involves: * Lowercasing the text. * Splitting text into words based on spaces. * Removing punctuation. * Mapping each unique word to a positive integer, creating a vocabulary index. * Truncating texts longer than 9406 words and padding shorter ones. * Word Embedding: The integer sequences are then fed into a Word Embedding layer. This layer maps each integer (representing a word) to a dense, multi-dimensional vector. The authors found an embedding dimensionality of 130 to be optimal.

Feature Extraction: The core of the model utilizes a two-layer Convolutional Neural Network (CNN) architecture combined with pooling layers for feature extraction.

  • First Convolutional Layer: A 1D CNN layer with 128 filters and a convolution window size of 72 is applied to the word embeddings. This layer extracts preliminary semantic features.
  • Max Pooling Layer: A Max Pooling layer follows, downsampling the feature maps and retaining the most important features within each channel. This reduces the dimensionality of the data.
  • Second Convolutional Layer: Another 1D CNN layer, this time with 64 filters and a convolution window size of 36, is used for deeper feature extraction.
  • Global Max Pooling Layer: A GlobalMax Pooling layer is applied to capture the most salient semantic information across the entire sequence.
  • Fully Connected Layer: Finally, a Fully Connected layer with a Sigmoid activation function performs the binary classification, outputting a probability indicating whether the user is an irony/stereotype spreader or not.

Datasets

The primary dataset used in this paper is derived from the "Profiling Irony and Stereotype Spreaders on Twitter" task organized by PAN at CLEF 2022.

  • Training Set: Consists of 420 XML files, each representing a single author and containing 200 tweets. This totals 84,000 tweets. A truth.txt file provides the ground truth labels ("I" or "NI") for these users.
  • Test Set: The paper mentions using a test set derived from the same PAN 2022 data. In one evaluation, they randomly split the 420 XML files into a training set of 320 files and a test set of 100 files. In another evaluation, they used the full 84,000 tweets for training and a separate test set of 180 users (each with a different number of comments).

Key Insights and Ideas

  • Effectiveness of CNNs for Text Classification: The paper reinforces the idea that CNNs, particularly 1D CNNs, are highly effective for extracting semantic features from text sequences, even for tasks like irony and stereotype detection.
  • Layered Feature Extraction: The hierarchical structure of the CNN with multiple convolutional and pooling layers allows for the extraction of increasingly abstract and important semantic features, moving from local word patterns to broader thematic elements.
  • Computational Efficiency: The authors highlight that their CNN-based approach is computationally less expensive and faster than some other complex neural network architectures (like recurrent neural networks) for similar tasks. This makes it a practical choice for real-world applications.
  • Importance of Preprocessing: The detailed text preprocessing steps, including word embedding, are crucial for preparing the text data in a format that the CNN can effectively learn from.

Key Results

  • Training Set Performance: The model achieved an accuracy of 0.8 and a loss of 0.3 on the training set. During training over 7 epochs, the accuracy on the training set reached 0.8143 with a loss of 0.3918.
  • Test Set Performance: When evaluated on a randomly split test set (100 users), the model achieved an accuracy of 0.85. The paper also states a running accuracy of 0.75 on the test set provided by PAN 2022.
  • Running Time: The model's training time was reported to be only 2 minutes.

Limitations and Future Work

  • Dataset Size: While the dataset is substantial, the authors do not explicitly discuss limitations related to potential biases or the representativeness of the data for all types of irony and stereotype spreaders.
  • Model Complexity: The paper suggests that for lightweight tasks, 1D CNNs can replace more complex networks. However, it doesn't explore the trade-offs in performance for more nuanced or complex linguistic phenomena.
  • Future Work: The paper implicitly suggests that further hyperparameter tuning and architectural exploration could lead to even better performance. The authors also mention that the model's ability to extract semantic features makes it suitable for other text classification tasks.

Summary updated

BibTeX
@inproceedings{yuan2022analysis,
  author = {Dong Yuan and Wenyin Yang and Li Ma and Zhenlin Tan and Qidi Lao},
  title = {{Analysis of Irony and Stereotype Spreaders Based On Convolutional Neural Networks}},
  booktitle = {CLEF (Working Notes)},
  year = {2022},
  url = {https://openreview.net/forum?id=ZA0J01nFAh},
}
Figure