Detailed Descriptions for Text Classification Applications

Gorka Artola, German RigauSEPLN Posters 2024 · OpenReview

Abstract

Training domain-specific text classifiers typically requires thousands of hand-labeled examples, a process that is both resource-intensive and slow. This research explores an alternative: fine-tuning language models on detailed class descriptions. By utilizing just 447 sentences describing the United Nations Sustainable Development Goals (SDGs), the authors demonstrate that a description-based model can outperform traditional baselines trained on over 8,000 manually labeled instances. The results suggest a more efficient, "human-like" path to achieving state-of-the-art performance in specialized classification tasks.

Introduction

In specialized domains such as law, medicine, or sustainability, building effective AI classifiers is often hindered by the "data hungry" nature of standard Pre-trained Language Models (PLMs). Typically, domain experts must manually label thousands of documents to teach models like BERT or GPT. This volume of expert-labeled data is often unavailable or prohibitively expensive to produce. While zero-shot models offer a potential solution, they frequently lack the precision required for complex, multi-faceted tasks.

This paper addresses this bottleneck by investigating whether detailed definitions of categories can serve as a viable substitute for massive lists of examples. The core question is: Can we leverage the semantic density of category descriptions to train models more effectively?

Methodology

The researchers propose a few-shot classification method framed as a Natural Language Inference (NLI) task. Instead of a standard "input-to-label" architecture, the problem is structured as a premise-hypothesis pair.

The classification task: Mapping a scientific paper title to relevant Sustainable Development Goals (SDGs).

The model, based on BART-Large MNLI, is asked to determine if a scientific paper title (the premise) entails a specific SDG description (the hypothesis). Training samples were generated by pairing description sentences with their corresponding SDG headlines (labeled as "entailment") and other SDG headlines (labeled as "contradiction"). This approach allows the model to learn the underlying semantic relationships rather than just surface-level patterns.

Datasets and Experimental Setup

The study utilized two primary datasets:

  1. SDG-Descriptions Dataset: A novel set of 447 sentences derived from official UN documentation, including SDG headlines, titles, targets, and indicators.
  2. Paper Titles Gold Dataset: A subset of the AURORA dataset, containing 9,382 expert-labeled scientific paper titles. This served as the training data for the baseline model and the evaluation benchmark for all experiments.

The performance of the description-based model was compared against a general baseline trained on 8,339 hand-labeled paper titles and a standard zero-shot model using only keywords (e.g., "No Poverty").

Key Results

The experimental results highlight the efficiency of description-based training:

  • Superior Performance with Less Data: The description-based model achieved a 67.12% F1-score using only 447 sentences. In contrast, the general baseline required nearly 3,000 hand-labeled samples to match this performance and trailed at 66.14% when trained on the full 8,339-sample set.
  • Surpassing Human Agreement: The best-performing iteration of the model reached an F1-score of 71.01%, exceeding the measured human expert agreement level of 70.10% for this task.
  • Value of Detail: Standard zero-shot classification using only keywords performed significantly worse (61.20%), underscoring that the "detailed" nature of the descriptions is critical for success.

Discussion and Limitations

The findings suggest that descriptions are "denser" than examples; high-quality semantic definitions help models generalize better than pattern-matching across noisy examples. This provides a "cold start" solution for researchers needing to build classifiers for new categories without existing labeled data.

However, some limitations were noted. The model struggled with specific goals like SDG 6 ("Clean water and sanitation") and SDG 17 ("Partnership for the goals"). Error analysis revealed that paper titles in these areas often use specialized language that does not align well with the public-policy language of UN descriptions. Furthermore, many "incorrect" predictions were found to be ambiguous cases where human experts also disagreed, suggesting the model is approaching the upper limits of the dataset's inherent clarity.

Conclusion

This work demonstrates that detailed class descriptions are a powerful shortcut to high-performance text classification. Future research may explore how generative Large Language Models (LLMs) can be used to further refine these descriptions, making them even more effective for training specialized classifiers.

Summary updated

BibTeX
@inproceedings{artola2024detailed,
  author = {Gorka Artola and German Rigau},
  title = {{Detailed Descriptions for Text Classification Applications}},
  booktitle = {SEPLN Posters},
  year = {2024},
  url = {https://openreview.net/forum?id=GSbMzaHXZx},
}
Figure