Improving aggressiveness detection using a data augmentation technique based on a Diffusion Language Model

Antonio D. Reyes-Ramírez, Mario Ezra Aragón, Fernando Sánchez-Vega, A. Pastor López-MonroyNAACL 2024

TL;DR

This paper tackles the challenge of detecting aggressive online content, which is often hampered by a lack of labeled data. They introduce a novel data augmentation technique using a Diffusion Language Model (DLM) to generate synthetic training data, showing it can improve aggressiveness detection models, especially when combined with other augmentation methods.

What Problem Does This Paper Address?

The proliferation of social media has led to a surge in online aggression, hate speech, and offensive language. Effectively detecting this harmful content is crucial for fostering healthier online communities. However, a major bottleneck in developing robust detection models is the scarcity of large, high-quality labeled datasets. Traditional supervised learning approaches, which are common for this task, heavily rely on such data. While data augmentation (DA) techniques exist to artificially expand datasets, many existing methods for text offer limited diversity and struggle to capture nuanced linguistic variations. This paper aims to address this data scarcity problem by proposing and evaluating a novel DA technique based on Diffusion Language Models, specifically for aggressiveness detection in Spanish tweets.

Methods

The core of this paper is a novel data augmentation technique leveraging a Diffusion Language Model (DLM). Specifically, they employ DiffuSeq, a non-autoregressive sequence-to-sequence model capable of conditional generation. This allows them to generate synthetic text samples that are conditioned on their respective class labels (aggressive or not aggressive).

The process involves:

  1. Training a Diffusion Language Model: The DLM is trained to generate synthetic text. To guide this generation, they create a specific dataset structure. They identify "offensive" words using chi-squared scores and then create pairs of tweets containing these words, along with their labels. The source sequence for the DLM is constructed to include a prompt for generating a tweet with a specific word and polarity, followed by an example tweet and its polarity. The target sequence is the actual tweet to be generated.
  2. Data Selection and Quality Assessment: The diffusion model can generate data of varying quality. The authors investigate this by sorting the generated synthetic data based on a confidence score from a base model (RoBERTuito). They then create "low," "middle," and "high-confidence" datasets to evaluate the impact of data quality on classifier performance.
  3. Classifier Training: A pre-trained language model, RoBERTuito, is fine-tuned on the original dataset and various augmented datasets to serve as the aggressiveness classifier.

They compare their DLM-based augmentation against two traditional DA techniques: Contextual Augmentation (using word substitution) and Easy Data Augmentation (EDA, which includes random swap, random delete, and synonym substitution).

Datasets

The primary dataset used in this study is the MEX-A3T dataset. This dataset consists of Mexican Spanish tweets labeled into two classes: "aggressive" and "not-aggressive." The paper provides a statistical breakdown of this dataset in Table 1, showing 5222 non-aggressive and 2110 aggressive tweets in the training set, and 2238 non-aggressive and 905 aggressive tweets in the test set. No new benchmark datasets were introduced; the focus was on augmenting an existing one.

Key Insights and Ideas

  • DLMs offer promising diversity: The paper highlights that DLMs, like DiffuSeq, have the potential to generate more diverse and class-conditional synthetic text compared to simpler DA methods.
  • Data quality matters: The authors found that not all generated data is equally beneficial. Augmenting with "middle-confidence" synthetic data from the DLM yielded the best results for their classifier.
  • Combinations are powerful: While individual DLM augmentation showed slight improvements, combining it with other methods, particularly synonym substitution, led to more robust performance. An ensemble of models trained on different augmentation strategies proved to be the most effective, though computationally expensive.
  • Nuance in aggression detection: The error analysis revealed that simple keyword matching isn't enough. Some messages containing offensive words might not be truly harmful, while others without explicit offensive words can still be aggressive, indicating the need for sophisticated contextual understanding.

Key Results

The study found that their DLM-based data augmentation, particularly using "middle-confidence" generated data, provided a slight but promising improvement in aggressiveness detection performance over the baseline (without augmentation). The best-performing classifier was achieved using the middle-confidence diff augmented data, reaching an F1-Macro score of 87.896±0.26 and an accuracy of 90.068±0.19.

When comparing their method to standard DA techniques:

  • Contextual augmentation (substitute) achieved F1-Macro of 87.6±0.61 and accuracy of 89.724±0.63.
  • EDA achieved F1-Macro of 87.402±0.57 and accuracy of 89.578±0.58.

The most significant gains were observed when combining their DLM augmentation with synonym substitution. An ensemble of models trained on different combinations of augmented datasets achieved the highest F1-Macro of 88.288±3.69 and accuracy of 90.322±4.59, though this was the most computationally expensive approach.

Limitations and Future Work

Limitations: * Dataset Labeling: The MEX-A3T dataset was manually labeled, meaning aggressiveness can be subjective and influenced by factors like gender, education, and cultural background. This inherent subjectivity can introduce biases. * Bias Propagation: Data augmentation techniques, including theirs, can propagate existing biases in the dataset. Their method, by conditioning on "bad words," propagates the bias that aggressiveness is strongly linked to offensive language. * Dataset Size: The dataset of 10,475 Spanish tweets is considered small for efficiently training a diffusion model, which has millions of parameters. They addressed this by pairing tweets.

Future Work: * Bias Analysis: Investigate potential sexism or gender bias within the MEX-A3T dataset and how models trained on it might acquire these biases. * Diversity Metrics: Employ more comprehensive metrics (e.g., Distinct-N, Self-BLUE) to quantitatively assess the diversity of synthetic data generated by the diffuser, alongside qualitative visual inspection. * Reducing Bias: Plan to reduce the bias of propagating "bad words" by increasing the number of tweets that do not contain offensive words. * Exploring DLMs: Envision their study as a catalyst for further research into DLM advantages for synthetic data generation and for hate speech detection in non-English languages.

Summary updated

BibTeX
@inproceedings{reyesramirez2024improving,
  author = {Antonio D. Reyes-Ramírez and Mario Ezra Aragón and Fernando Sánchez-Vega and A. Pastor López-Monroy},
  title = {{Improving aggressiveness detection using a data augmentation technique based on a Diffusion Language Model}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2024},
}
Figure