FilipN@LT-EDI-ACL2022-Detecting signs of Depression from Social Media: Examining the use of summarization methods as data augmentation for text classification

Filip Nilsson, György KovácsACL 2022

TL;DR

This paper explores using abstractive summarization with T5 to augment imbalanced social media datasets for depression detection. They found that strategically increasing the minority class size through summarization can improve BERT's classification performance, offering a practical approach to tackle class imbalance in sensitive NLP tasks.

What Problem Does This Paper Address?

Depression is a significant mental health issue, and early detection is crucial for effective treatment. Social media offers a vast, albeit noisy, source of data that could potentially reveal signs of depression. However, training machine learning models, particularly powerful transformer-based ones like BERT, on this data is hampered by class imbalance. Many social media datasets for depression detection are heavily skewed, with far fewer examples of depression-related posts compared to neutral ones. This imbalance can lead to models that perform poorly on the minority (depression) class, failing to identify those most in need of help. This paper aims to address this class imbalance problem by investigating data augmentation techniques specifically for this domain.

Methods

The core of this paper's methodology lies in using abstractive summarization as a data augmentation technique. They leverage Google's T5 model, pre-trained on the c4 dataset and fine-tuned for abstractive summarization. The process involves:

  1. Summarization: For each post in the training dataset, T5 generates a summary. The paper notes that the length of these summaries is a parameter that can be tuned (they experimented with lengths between 10 and 50 tokens).
  2. Data Augmentation: The generated summaries are then used to augment the training data. The goal is to increase the representation of the underrepresented classes (severe and moderate depression) to balance the dataset.
  3. Classification: A BERT base model is then fine-tuned on these augmented datasets. The authors experimented with five different degrees of balancing, where "50" signifies that the minority classes are at least 50% as large as the largest class.

The paper highlights that while abstractive summarization is the chosen method, extractive summarization is also a common approach in NLP. They also mention that the T5 model's maximum sequence length of 512 tokens can be a limitation, potentially causing it to miss crucial information in longer posts.

Datasets

The primary dataset used in this study is the LT-EDI-ACL2022 dataset. This dataset contains social media posts categorized into "severe," "moderate," and "not depression." The authors note that this dataset is relatively small and significantly imbalanced, making it a suitable candidate for studying data augmentation. They were provided with training, validation, and test sets, but the test set labels were not published at the time of writing, so the validation set was used as the test set for their experiments.

Key Insights and Ideas

  • Strategic Augmentation is Key: Simply augmenting the data to achieve perfect balance isn't always optimal. The paper suggests that there's a "sweet spot" where increasing the minority class size through summarization can significantly improve performance without drastically hurting precision.
  • Summarization Captures Sentiment (Sometimes): While T5's summaries can be effective, the authors observed that they don't always capture the most critical sentiment-bearing parts of the original post. This highlights the inherent challenge of summarization and its potential limitations in data augmentation.
  • BERT Remains a Strong Baseline: The study uses BERT as the classification model, reinforcing its status as a robust choice for text classification tasks, even when dealing with augmented data.

Key Results

The experiments showed that data augmentation using T5 summarization had a positive impact on the BERT model's performance. Specifically:

  • Improved Macro F1-Score: The Macro F1-score, a metric that balances precision and recall across all classes, saw an increase from 0.50 (no augmentation) to a peak of 0.52 at the 25%, 50%, and 75% balancing degrees.
  • Recall Boost: Macro Recall also improved, reaching 0.54 at the 50% balancing degree.
  • Precision Trade-off: While recall improved, precision saw a slight decrease at higher augmentation levels, indicating a trade-off that needs careful management. The authors suggest that selecting the right degree of balancing can improve recall without a significant negative effect on precision.

Limitations and Future Work

The authors acknowledge several limitations and propose future research directions:

  • Summarization Length: The impact of different summary lengths (10-50 tokens) was explored, but further investigation into optimal lengths and strategies for handling posts longer than T5's 512-token limit (e.g., splitting and concatenating summaries) is suggested.
  • Extractive vs. Abstractive: The paper focused on abstractive summarization. Future work could compare its effectiveness against extractive methods like MatchSum.
  • Dataset and Domain Generalization: The study was conducted on a single dataset. Applying this augmentation technique to different domains and datasets is crucial to assess its generalizability.
  • Alternative Transformer Models: While BERT was used, exploring other transformer architectures for classification could reveal different performance characteristics with this augmentation method.
  • Beyond Summarization: The authors suggest exploring other data augmentation techniques beyond summarization to further improve performance.

Summary updated

BibTeX
@inproceedings{nilsson2022filipn,
  author = {Filip Nilsson and György Kovács},
  title = {{FilipN@LT-EDI-ACL2022-Detecting signs of Depression from Social Media: Examining the use of summarization methods as data augmentation for text classification}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2022},
}
Figure