Ustnlp16 at SemEval-2025 Task 9: Improving Model Performance through Imbalance Handling and Focal Loss

Zhuoang Cai, Zhenghao Li, Yang Liu, Liyuan Guo, Yangqiu Song

TL;DR

This paper tackles the challenge of accurately classifying food hazards from short, imbalanced text reports. The authors demonstrate that combining data augmentation techniques like Easy Data Augmentation (EDA) with focal loss significantly improves model performance, especially for rare but critical hazard categories.

What Problem Does This Paper Address?

Accurately identifying food hazards from text is crucial for public safety. However, this task is fraught with challenges:

  • Severe Class Imbalance: Some food hazards are reported far more frequently than others, leading standard models to favor majority classes and miss critical, rare events.
  • Short and Unstructured Text: Food recall reports are often brief, lack formal structure, and can contain domain-specific jargon, making them difficult for NLP models to parse.
  • Overlapping Semantic Categories: Different hazards might be described using similar language, further complicating classification.

Traditional methods like manual inspection are slow and error-prone. While transformer-based models like BERT and RoBERTa have advanced NLP, they still struggle with imbalanced datasets. This paper aims to improve their performance on this specific, real-world problem.

Methods

The researchers employed a multi-pronged approach to address class imbalance and improve classification:

  • Transformer-Based Models: They used BERT and RoBERTa as the backbone for their text classification models, leveraging their strong contextual understanding capabilities.
  • Data Augmentation (EDA): Easy Data Augmentation (EDA) was a key technique. It involves simple text manipulations like synonym replacement, random insertion, random swap, and random deletion. This helps create more diverse training examples, particularly for minority classes.
  • Random Oversampling: This technique artificially increases the number of instances for minority classes by duplicating existing samples until they reach a target proportion of the majority class.
  • Focal Loss: Instead of standard cross-entropy loss, they used focal loss. This modified loss function down-weights easy-to-classify examples and focuses more on hard-to-classify ones, which are often the minority class instances.

The authors explored combinations of these techniques, applying EDA before tokenization, oversampling after tokenization, and focal loss during training.

Datasets

The primary dataset used in this study is from SemEval-2025 Task 9: Food Hazard Detection. This dataset consists of thousands of short food recall reports obtained from government agencies. It is characterized by:

  • Short text descriptions: Titles and brief recall identifiers.
  • Unstructured format: Varying lengths and styles.
  • Severe class imbalance: Many hazard categories are significantly underrepresented.

The paper also mentions using a "product" prediction task (ST2) and a "hazard" prediction task (ST1) from the SemEval challenge.

Key Insights and Ideas

  • EDA is a Powerful Standalone: The study found that Easy Data Augmentation (EDA) alone provided significant improvements in handling class imbalance, often outperforming combinations with other methods. This highlights the effectiveness of simple text perturbations for boosting minority class representation.
  • Synergy is Present, but Nuanced: While combining oversampling and focal loss with EDA generally improved performance over baseline models, these combinations didn't always surpass EDA alone. This suggests that the specific way these techniques interact is crucial and might require further tuning.
  • Focal Loss for Hard Cases: Focal loss proved valuable for focusing on difficult-to-classify examples, which are critical for improving recall on minority classes.
  • Transformer Models Benefit Greatly: The inherent power of BERT and RoBERTa, when combined with effective imbalance handling, leads to substantial gains in accuracy and F1 scores for this challenging task.

Key Results

The paper reports significant improvements across various configurations:

  • Product Detection (ST2):
    • The baseline BERT model achieved an accuracy of 0.22 and an F1-Macro score of 0.03.
  • Oversampling with a rate of 0.1 improved accuracy to 0.50 and F1-Macro to 0.25.
  • EDA with a rate of 0.2 achieved the highest accuracy of 0.55 and an F1-Macro score of 0.30.
  • Hazard Detection (ST1):
    • The combination of focal loss and EDA (0.1 rate) with BERT achieved an accuracy of 0.86 and an F1-Macro score of 0.59.
  • RoBERTa with the same configuration performed comparably.

The authors' system ranked 13th on ST1 and 12th on ST2 in the SemEval-2025 competition.

Limitations and Future Work

Limitations:

  • Synthetic Noise: EDA, while effective, might introduce synthetic noise into the dataset, potentially leading to misclassifications if augmented samples don't perfectly retain semantic meaning.
  • Computational Cost: Transformer models are computationally intensive, making real-time applications or deployment in resource-constrained environments challenging.
  • Hyperparameter Tuning: Focal loss requires careful hyperparameter tuning, which might not generalize well across different datasets or tasks.
  • Generalizability: The model was trained on a specific dataset (SemEval-2025 Task 9). Its performance on other food safety datasets with different linguistic characteristics or reporting styles might be limited.
  • Majority-Minority Gap: Despite improvements, a performance gap between majority and minority classes still exists.

Future Work:

  • Domain Adaptation: Explore techniques to adapt the model to different domains and datasets.
  • Advanced Augmentation: Investigate more sophisticated data augmentation methods.
  • Refined Loss Functions: Explore other advanced loss functions.
  • Fairness and Robustness: Investigate techniques like contrastive learning, cost-sensitive training, or adaptive resampling to further improve model fairness and robustness.

Summary updated

Abstract

Classification tasks often suffer from imbal- anced data distribution, which presents chal- lenges in food hazard detection due to severe class imbalances, short and unstructured text, and overlapping semantic categories. In this paper, we present our system for SemEval- 2025 Task 9: Food Hazard Detection, which ad- dresses these issues by applying data augmenta- tion techniques to improve classification perfor- mance. We utilize transformer-based models, BERT and RoBERTa, as backbone classifiers and explore various data balancing strategies, including random oversampling, Easy Data Augmentation (EDA), and focal loss. Our ex- periments show that EDA effectively mitigates class imbalance, leading to significant improve- ments in accuracy and F1 scores. Furthermore, combining focal loss with oversampling and EDA further enhances model robustness, par- ticularly for hard-to-classify examples. These findings contribute to the development of more effective NLP-based classification models for food hazard detection.
BibTeX
@inproceedings{cai2025ustnlp,
  author = {Zhuoang Cai and Zhenghao Li and Yang Liu and Liyuan Guo and Yangqiu Song},
  title = {{Ustnlp16 at SemEval-2025 Task 9: Improving Model Performance through Imbalance Handling and Focal Loss}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2025},
}
Figure