Dolomites@#SMM4H 2024: Helping LLMs “Know The Drill” in Low-Resource Settings - A Study on Social Media Posts

Giuliano Tortoreto, Seyed Mahed MousaviACL 2024

TL;DR

This paper introduces two novel multi-task learning data augmentation techniques, "In-domain Augmentation" and "Drills," to improve the performance of Large Language Models (LLMs) on low-resource tasks. By leveraging related data and extracting smaller sub-tasks, these methods significantly boost performance on social media text classification, entity recognition, and information extraction tasks, demonstrating the power of knowledge transfer in data-scarce scenarios.

What Problem Does This Paper Address?

Fine-tuning Large Language Models (LLMs) for specific downstream tasks often requires substantial amounts of labeled data. However, acquiring such data can be expensive and time-consuming, especially for specialized domains or low-resource languages. This poses a significant challenge when deploying LLMs in real-world scenarios where data availability is limited. Traditional data augmentation techniques help, but this paper explores a more sophisticated approach rooted in multi-task learning (MTL) to enhance LLM performance in these data-constrained environments. The authors specifically target tasks within the SMM4H 2024 competition, which often involve analyzing social media data, a domain known for its noisy and often limited labeled datasets.

Methods

The paper proposes two novel Multi-Task Learning Data Augmentation (MTL-DA) techniques:

  • In-domain Augmentation: This method augments the target task's dataset with data from other related tasks that share the same data source (in this case, Reddit). The LLM is then jointly fine-tuned on this combined dataset, allowing it to learn from a broader, yet related, set of information.
  • Drills: This technique involves extracting smaller, more focused sub-tasks from the main target task. The training data is then replicated to include examples for these "drills." For instance, in an entity recognition task, drills might focus on identifying specific types of entities or their relationships. The LLM is then trained on the original task data along with these drill tasks.

The authors experimented with the Mistral 7B model, fine-tuning it using QLoRA for memory efficiency. They evaluated their approaches on three SMM4H 2024 tasks: multi-class classification of social anxiety symptoms, entity recognition for clinical and social impacts, and self-reported age classification.

Datasets

The study utilizes datasets from the SMM4H 2024 competition for three tasks:

  • Task 3 (Social Anxiety Classification): Approximately 3000 annotated Reddit posts, split into training, validation, and test sets.
  • Task 4 (Clinical/Social Impact Extraction): Approximately 1380 annotated Reddit posts for entity recognition, with training, validation, and test sets.
  • Task 6 (Self-Reported Age Classification): A cross-platform dataset including tweets and Reddit posts, with a significant amount of training data (8.8k tweets, 100k Reddit posts) and smaller validation/test sets.

No new datasets were introduced; the focus was on augmenting existing ones.

Key Insights and Ideas

  • Leveraging Domain Similarity is Key: The "In-domain Augmentation" approach highlights that even if auxiliary tasks are different, their shared origin (e.g., Reddit) provides a valuable signal for improving performance on the target task.
  • Decomposition Enhances Learning: The "Drills" method demonstrates that breaking down a complex task into smaller, manageable sub-tasks can significantly improve an LLM's ability to learn and generalize, especially when combined with in-domain data.
  • Fine-tuning Outperforms In-Context Learning (ICL) for Precision: While ICL with powerful models like GPT-4 achieved high recall, the fine-tuned Mistral model, especially with augmentation, managed to achieve higher precision and, consequently, superior F1 scores. This suggests that for low-resource settings where data is scarce, fine-tuning with effective augmentation can be more beneficial than relying solely on prompt engineering.
  • Augmentation Improves Robustness: The paper suggests that these augmentation techniques not only boost performance but also make models more robust to unseen data, as evidenced by performance on the test sets.

Key Results

  • Both "In-domain Augmentation" and "Drills" techniques consistently improved the F1 scores on the validation sets for Tasks 3 and 4 compared to vanilla fine-tuning.
  • The "In-domain Augmentation + Drills" approach achieved the highest F1 score for Task 4 and the highest score for Task 3 when combined with additional drills.
  • On the SMM4H test sets, the "In-domain Augmentation + Drills" model scored two to five points higher than the overall mean scores of the competition participants for Tasks 3 and 4.
  • For Task 6, while vanilla fine-tuning performed best on the validation set, the "In-domain Augmentation + Drills" model outperformed it on the test set, suggesting improved generalization.

Limitations and Future Work

The authors acknowledge several open questions for future research:

  • Optimal Drill Selection: What constitutes the "best set of drills" for a given task?
  • Defining "In-domain": How can we formally define when a dataset is considered "in-domain"?
  • Cross-Domain Transfer: Would tasks from different domains also contribute positively to performance?
  • Comparison to Existing Methods: How does their proposed approach compare to other established data augmentation techniques?
  • Overfitting Risk: Does replicating the dataset for drills potentially lead to overfitting?

The authors plan to explore these questions in future work.

Summary updated

BibTeX
@inproceedings{tortoreto2024dolomites,
  author = {Giuliano Tortoreto and Seyed Mahed Mousavi},
  title = {{Dolomites@\#SMM4H 2024: Helping LLMs “Know The Drill” in Low-Resource Settings - A Study on Social Media Posts}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
}
Figure