Description Boosting for Zero-Shot Entity and Relation Classification

Gabriele Picco, Leopold Fuchs, Marcos Martínez Galindo, Alberto Purpura, Vanessa López, Hoang Thanh Lam

Zero-shot learning (ZSL) has emerged as a powerful paradigm in Natural Language Processing (NLP), enabling models to categorize text into classes unseen during training. This capability hinges on the use of textual descriptions to define category semantics. However, recent research highlights a critical vulnerability: zero-shot models are highly sensitive to the specific phrasing of these descriptions. A minor lexical change—such as substituting "Company" for "Organization"—can significantly shift the model's decision boundaries, leading to inconsistent performance.

The Challenge: Description Sensitivity

The primary bottleneck in current ZSL workflows is the manual effort required to craft "optimal" descriptions. For non-experts, this often becomes a process of trial and error, as the internal logic of black-box models is difficult to predict.

Figure 1: Illustration of description sensitivity. Changing the description of 'Company' or 'Fruits' alters the model's classification of the word 'Apple'.

As shown in Figure 1, the same mention can be classified differently based on the provided context and class definitions. This sensitivity gap suggests that zero-shot performance is as much a function of input semantics as it is of model architecture.

The UDEBO Framework

To address this, we introduce UDEBO (Unsupervised DEscription BOosting), an unsupervised framework designed to automate the generation and selection of diverse class descriptions. UDEBO operates in two distinct stages:

1. Automated Description Generation

UDEBO leverages four distinct strategies to create a diverse pool of descriptions, ensuring the model captures various semantic nuances:

  • Pre-trained LMs: GPT-2 is utilized to expand brief descriptions into more comprehensive narratives.
  • Fine-tuned LMs: A T5 model, fine-tuned on millions of Wikidata instances, generates structured and detailed definitions.
  • Summarization: A BERT2BERT model distills noisy descriptions into concise, core definitions.
  • Paraphrasing: A Pegasus model simplifies linguistic structures to improve generalizability.

2. Ranking and Ensembling

Once a set of candidate descriptions is generated, UDEBO employs an unsupervised ranking mechanism based on Information Entropy. Descriptions that result in lower entropy—indicating higher model confidence—are prioritized. Finally, the framework uses a majority voting ensemble across multiple classification pipelines to determine the final label, effectively triangulating the correct category through diverse semantic perspectives.

Key Results

UDEBO was evaluated across four benchmark datasets: OntoNotesZS and MedMentionsZS for entity classification, and FewRel and WikiZS for relation classification. The framework achieved new state-of-the-art (SOTA) results, demonstrating substantial improvements over existing baselines.

Table 1: Performance comparison on FewRel and WikiZS datasets. UDEBO consistently outperforms individual description strategies and the ZS-BERT baseline.

As detailed in Table 1, UDEBO significantly outperforms individual strategies and the standard ZS-BERT model across all metrics. Notable gains include a 7 percentage point improvement in Macro F1 on OntoNotesZS and a 6 percentage point boost on FewRel. While gains in specialized domains like MedMentionsZS were more modest (1.3 percentage points), the overall trend confirms that description ensembling is a robust strategy for zero-shot tasks.

Discussion and Conclusion

The success of UDEBO underscores the principle that diversity in input descriptions often provides more value than the precision of any single prompt. By using entropy as an unsupervised proxy for quality, UDEBO allows smaller, more efficient models to achieve performance levels previously reserved for massive, computationally expensive LLMs. Future work will focus on reducing the computational overhead of running multiple inference pipelines and exploring more sophisticated generation techniques to further refine the description boosting process.

Summary updated

Abstract

Zero-shot entity and relation classification models leverage available external information of unseen classes – e.g., textual descriptions – to annotate input text data. Thanks to the minimum data requirement, Zero-Shot Learning (ZSL) methods have high value in practice, especially in applications where labeled data is scarce. Even though recent research in ZSL has demonstrated significant results, our analysis reveals that those methods are sensitive to provided textual descriptions of entities (or relations). Even a minor modification of descriptions can lead to a change in the decision boundary between entity (or relation) classes. In this paper, we formally define the problem of identifying effective descriptions for zero shot inference. We propose a strategy for generating variations of an initial description, a heuristic for ranking them and an ensemble method capable of boosting the predictions of zero-shot models through description enhancement. Empirical results on four different entity and relation classification datasets show that our proposed method outperform existing approaches and achieve new SOTA results on these datasets under the ZSL settings. The source code of the proposed solutions and the evaluation framework are open-sourced.
BibTeX
@inproceedings{picco2024description,
  author = {Gabriele Picco and Leopold Fuchs and Marcos Martínez Galindo and Alberto Purpura and Vanessa López and Hoang Thanh Lam},
  title = {{Description Boosting for Zero-Shot Entity and Relation Classification}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
  doi = {10.18653/v1/2024.findings-acl.562},
}
Figure