WADER at SemEval-2023 Task 9: A Weak-labelling framework for Data augmentation in tExt Regression Tasks

Manan Suri, Aaryak Garg, Divya Chaudhary, Ian Gorton, Bijendra KumarACL 2023

TL;DR

This paper introduces WADER, a novel weak-labeling framework for data augmentation in text regression tasks, specifically addressing data imbalance and scarcity. WADER leverages translation and a clever sampling strategy to generate synthetic data, proving effective for cross-lingual, zero-shot scenarios and improving performance on the SemEval-2023 Task 9 for multilingual tweet intimacy analysis.

What Problem Does This Paper Address?

Analyzing textual intimacy is crucial for understanding human relationships and social norms, but it's a challenging task computationally. Real-world datasets for such tasks often suffer from two major issues: data imbalance, where certain intimacy levels are underrepresented, and data scarcity, meaning there simply isn't enough labeled data. This is particularly problematic for multilingual tasks where data for less common languages or specific intimacy levels might be scarce. Existing data augmentation techniques can be insufficient, leading to models that don't generalize well, especially in cross-lingual or zero-shot settings where models must perform on languages they haven't seen during training.

Methods

The core of WADER is its weak-labeling strategy for data augmentation. It tackles data imbalance and scarcity through a two-pronged approach:

  1. Distribution-based Sampling: Instead of augmenting all data, WADER selectively samples tweets based on their label distribution. It sets a threshold (e.g., 3.2 on a 1-5 scale) and focuses augmentation efforts on tweets with labels above this threshold, aiming to bolster underrepresented categories.
  2. Translation-based Augmentation: For selected tweets, WADER employs translation. For seen languages, it translates tweets to other languages and then back to the original, creating paraphrased versions. For unseen languages, it directly translates sampled tweets from seen languages. This process generates synthetic data that increases linguistic diversity.
  3. Label Validation: To ensure the quality of augmented data, a baseline model is used to infer labels on the translated corpus. A "difference-based sampling" method then uses the absolute difference between the predicted and original labels to filter out low-quality translations, using a threshold $\beta$.

The augmented data is then used to fine-tune state-of-the-art pre-trained language models like XLM-RoBERTa and XLNET. Finally, an ensemble of these models is used for prediction.

Datasets

The primary dataset used is the MINT- Multilingual INTimacy analysis (MINT) dataset for SemEval-2023 Task 9. This dataset contains tweets in 6 languages (English, Spanish, Italian, Portuguese, French, and Chinese) for training, and evaluation is performed on these seen languages as well as 4 unseen languages (Hindi, Arabic, Dutch, and Korean). The intimacy is annotated on a 5-point Likert scale. The paper does not introduce a new benchmark dataset.

Key Insights and Ideas

  • Targeted Augmentation is Key: Simply augmenting all data isn't optimal. WADER's distribution-based sampling, focusing on underrepresented labels, is crucial for addressing imbalance effectively.
  • Translation as a Powerful Augmentation Tool: Back-translation and cross-lingual translation prove to be effective in generating diverse yet semantically relevant synthetic data, especially for multilingual and zero-shot scenarios.
  • Quality Control Matters: The label validation step, using difference-based sampling, is vital to filter out noisy augmented data and prevent performance degradation.
  • Multilingual Pre-training is Essential: XLM-RoBERTa significantly outperforms XLNET on most languages, highlighting the importance of models pre-trained on diverse multilingual corpora for tasks involving multiple languages, especially those with non-Latin scripts.

Key Results

  • WADER achieved a rank of 32nd overall in the SemEval-2023 Task 9 competition, with 34th on seen languages and 29th on unseen languages.
  • The proposed WADER framework outperforms baseline models that use only gold-labeled data.
  • Moderate values of the difference sampling threshold $\beta$ (0.1, 0.2) yielded better results than very low values, suggesting that a balance between data quality and corpus size is beneficial. This is hypothesized to provide regularization and prevent overfitting.
  • XLM-RoBERTa-based models generally performed better than XLNET-based models across most languages, especially for Hindi and Korean where XLNET showed negative R coefficients.

Limitations and Future Work

The authors acknowledge several limitations:

  • Translation Quality: The performance of WADER is heavily dependent on the quality of machine translation. For languages with non-Latin scripts (like Hindi), translation quality can be lower, impacting augmentation effectiveness.
  • Overfitting: While translation increases diversity, sentences can remain semantically similar, potentially leading to overfitting. The authors note that higher difference thresholds (leading to potentially more diverse but lower quality translations) sometimes yielded better results, suggesting a trade-off.
  • Word Sensitivity: Textual intimacy relies on specific vocabulary. Translations might replace words that are crucial for conveying intimacy, thus altering the semantic nuance.

Future work could explore more advanced translation quality assessment and refinement techniques, investigate methods to better capture word sensitivity in translations, and explore other data augmentation strategies beyond translation.

Summary updated

BibTeX
@inproceedings{suri2023wader,
  author = {Manan Suri and Aaryak Garg and Divya Chaudhary and Ian Gorton and Bijendra Kumar},
  title = {{WADER at SemEval-2023 Task 9: A Weak-labelling framework for Data augmentation in tExt Regression Tasks}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
}
Figure