CICLe: Conformal In-Context Learning for Largescale Multi-Class Food Risk Classification

Korbinian Randl, John Pavlopoulos, Aron Henriksson, Tony LindgrenACL 2024 · DOI

Introduction

Rapid response to food safety incidents, such as contamination or mislabeling, is critical for public health. While food recall announcements are publicly available, they are often unstructured, multilingual, and span thousands of diverse product and hazard categories. Automating the classification of these announcements presents a significant "long-tail" challenge: common hazards like Salmonella are well-represented, while rare chemical contaminants have very few examples. Traditional machine learning (ML) models are efficient but struggle with low-support classes, whereas Large Language Models (LLMs) possess strong reasoning capabilities but are computationally expensive and can be overwhelmed by large label spaces.

Methodology: The CICLe Framework

To address these challenges, we introduce CICLe (Conformal In-Context Learning), a hybrid "LLM-in-the-loop" framework designed to combine the efficiency of traditional ML with the reasoning power of LLMs. The process involves three main stages:

  1. Base Classification: A lightweight model, such as Logistic Regression with TF-IDF, performs an initial classification.
  2. Conformal Prediction (CP): Instead of a single point estimate, the system generates a prediction set—a subset of classes statistically guaranteed to contain the true label at a predefined confidence level (e.g., 95%).
  3. Adaptive Prompting:
    • If the base classifier is highly certain (prediction set size = 1), the LLM is bypassed.
  4. If uncertain, the small subset of candidate classes is passed to an LLM as a few-shot prompt.

This approach effectively filters noise and reduces the classification task from a search across over 1,000 classes to a manageable multiple-choice problem for the LLM.

Dataset Characteristics

The study introduces a comprehensive benchmark dataset comprising 7,546 short text titles from food recall announcements across 24 international domains. The dataset features multi-level labeling for both food products and hazards at coarse and fine granularities.

Distribution of hazard and product categories across the dataset, highlighting the long-tail nature of the classification task.

The dataset is also multilingual, including samples in English, German, French, Greek, Italian, and Danish, reflecting the global nature of food supply chains.

Temporal distribution of food recall announcements by language from 1995 to 2022.

Key Results and Insights

Our evaluation revealed several critical findings:

  • Performance Gains: On the fine-grained product classification task (1,256 classes), CICLe achieved an $F_1$ score of 0.58, significantly outperforming the base TF-IDF-LR model (0.54).
  • Efficiency: By routing only uncertain cases to the LLM, CICLe reduced LLM calls by up to 40% in simpler tasks without compromising accuracy.
  • Model Robustness: Traditional models like SVMs and Logistic Regression surprisingly outperformed Transformers on extremely low-support classes, likely due to the keyword-heavy nature of recall titles. CICLe leverages LLM world knowledge to "boost" performance precisely where these base models falter.

Conclusion and Future Work

While CICLe demonstrates significant improvements in food risk classification, challenges remain regarding label noise—where titles may lack specific hazard information found in full articles—and the inherent trade-off between prediction set size and accuracy guarantees. Future research will explore the integration of smaller, open-source LLMs like Llama 3 into the CICLe framework to further enhance accessibility and efficiency.

The researchers have released the dataset and code publicly, providing a vital resource for the intersection of NLP and public health.

Summary updated

BibTeX
@inproceedings{randl2024cicle,
  author = {Korbinian Randl and John Pavlopoulos and Aron Henriksson and Tony Lindgren},
  title = {{CICLe: Conformal In-Context Learning for Largescale Multi-Class Food Risk Classification}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
  doi = {10.18653/v1/2024.findings-acl.459},
}
Figure