UniBuc at SemEval-2025 Task 9: Similarity Approaches to Classification

Marius Micluta-CampeanuACL 2025

TL;DR

This paper introduces an unsupervised, similarity-based approach for classifying food safety hazards and products from incident reports. By leveraging semantic label properties and avoiding traditional training, the method offers improved interpretability and handles ambiguous labels, achieving a respectable 14th place in the SemEval-2025 Task 9.

What Problem Does This Paper Address?

The global food industry faces increasing complexity in its supply chains, making it challenging to monitor and control food safety hazards. Natural Language Processing (NLP) offers a powerful way to analyze the vast amounts of text data generated by food-incident reports. However, existing classification systems often struggle with noisy data, imbalanced class distributions, and a lack of interpretability. The SemEval-2025 Task 9 specifically aims to address the explainability of these classification systems. This paper tackles the challenge of accurately identifying food hazards and products from incident reports, particularly when dealing with ambiguous labels and the need for transparent, understandable predictions.

Methods

The core of the proposed system is an unsupervised, similarity-based classification approach. Instead of training a model on labeled data, the system leverages the semantic properties of the labels themselves.

Here's a breakdown of the key steps:

  • Data Cleaning: This crucial initial step involves removing duplicate lines, HTML markup, and irrelevant instructions from the raw text. A Large Language Model (LLM) is then used to extract the hazard or product, along with a brief description.
  • Pre-processing: Task-specific stop words are removed. For hazard classification, lemmatization is applied to both texts and labels, which helps in reducing word variations.
  • Similarity Classification: The system models classification as a similarity search problem. Texts and labels are embedded using Sentence Transformers models. Cosine similarity is then computed between the embedded text and each embedded label. The label with the highest similarity score is chosen. The LLM-generated descriptions aid this process by including more common terms.
  • Post-processing (for Hazards): To handle cases where an incident report might involve multiple hazards, a prioritization strategy is employed. Hazards are ranked by importance and specificity, with higher-risk categories (like allergens) being prioritized.

A separate pipeline is used for product classification, omitting some components that did not improve predictions. Notably, the entire process involves no training or fine-tuning of models.

Datasets

The primary dataset used is from the SemEval-2025 Task 9: The Food Hazard Detection Challenge. This dataset consists of food-incident reports written in English, which are a subset of a larger dataset that also includes texts in German and other languages. The task has two sub-tasks:

  1. Predicting coarse-grained categories for hazards and products.
  2. Predicting the exact hazard and product.

The authors participated only in the second sub-task.

Key Insights and Ideas

  • Unsupervised Learning for Explainability: The paper strongly advocates for unsupervised, similarity-based methods as a path to more interpretable classification systems, especially in high-stakes domains like food safety.
  • Leveraging Label Semantics: The core idea is that the meaning embedded within the labels themselves can be powerful enough for classification, bypassing the need for extensive labeled training data.
  • Handling Ambiguity: The similarity-based approach inherently handles ambiguous labels better than direct mapping, as similar but distinct labels can still be identified. The authors note that the dataset's design choice to include ambiguous labels was a significant challenge.
  • LLMs for Data Augmentation and Cleaning: LLMs are used not just for classification but also for crucial pre-processing steps like extracting descriptions and cleaning noisy text, demonstrating their versatility.
  • Prioritizing Risk: The post-processing step for hazard classification highlights the importance of not just identifying hazards but also prioritizing them based on their potential risk to human health.

Key Results

  • The team ranked 14th out of 26 teams in the second sub-task of SemEval-2025 Task 9.
  • They achieved an F1-score of 0.3453 for the overall second sub-task.
  • Specifically, they obtained an F1-score of 0.4257 for 128 hazards and 0.2528 for 1142 products.
  • The system is described as unsupervised, employing small models, needing modest resources, and allowing for easy interpretation.
  • The authors found that larger embedding models did not always lead to improved scores, suggesting a trade-off between model size and performance for this task.

Limitations and Future Work

  • Error Accumulation: As a cascading system, errors in early stages (like data cleaning) can propagate and negatively impact later stages.
  • Negation Handling: The similarity approach struggles with negations, potentially leading to false positives.
  • High-Level Concept Understanding: The system has difficulty capturing high-level concepts, such as discerning brands from products or understanding underlying causes.
  • Model Independence: Most experiments were conducted using off-the-shelf models without training. The viability of the method with other models or fine-tuned embeddings remains to be explored.
  • Label Overlap: The significant label overlap in the dataset remains a challenge, particularly for product classification.

Future work suggested includes:

  • Fine-tuning embedding models: Leveraging training data labels to enhance embeddings.
  • Improving data cleaning: Developing more robust techniques for cleaning the input text.
  • Mitigating similarity limitations: Exploring new techniques to overcome the inherent limitations of similarity-based methods.
  • Specialized Corpora: Investigating the use of specialized corpora like FoodBase for enhanced performance.
  • Dataset Redesign: The authors suggest that the task itself could benefit from a redesign to address significant label overlap.

Summary updated

BibTeX
@inproceedings{miclutacampeanu2025unibuc,
  author = {Marius Micluta-Campeanu},
  title = {{UniBuc at SemEval-2025 Task 9: Similarity Approaches to Classification}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2025},
}
Figure