A Simple and Effective Framework for Strict Zero-Shot Hierarchical Classification

Rohan Bhambhoria, Lei Chen, Xiaodan Zhu

Abstract

Hierarchical classification in the "long-tail" distribution presents a significant challenge for zero-shot learning. While Large Language Models (LLMs) possess vast world knowledge, they often struggle with large label spaces and specific hierarchical constraints. This paper introduces Primed+, a framework that treats the LLM as a noisy knowledge graph to augment a specialized Entailment Predictor. By utilizing a composite function approach—$E(L(E(X)))$—the framework achieves superior performance on complex datasets like Web of Science and Amazon Beauty without any parameter updates.

Introduction

In real-world data environments, information is rarely distributed evenly. E-commerce platforms and academic databases typically exhibit a "long-tail" distribution, where a few broad categories contain the majority of instances, while thousands of niche, specific categories (leaf nodes) contain very few. Standard Large Language Models (LLMs) frequently falter in these environments when tasked with classifying text into specific hierarchical labels.

The authors identify three primary reasons for this failure: the rarity of specific labels in training data, the distraction caused by excessively large label spaces, and the potential for input text to exceed the model's context window. Current benchmarks often overlook these "strict zero-shot" challenges, where a model must perform on a novel task without any prior examples or parameter fine-tuning. This paper proposes a novel framework to bridge the gap between general LLM knowledge and precise classification logic.

The Primed+ Framework

The core of the proposed solution is a "composite function" approach that reframes the LLM's role. Rather than acting as the final classifier, the LLM serves as a noisy knowledge graph that provides context to a more specialized model. The framework, termed Primed+, utilizes two distinct components:

  1. The Entailment Predictor ($E$): A smaller, specialized model (such as BART-MNLI) trained to determine if one sentence logically implies another. While excellent at selecting from a closed list of labels, it lacks the deep world context required for complex reasoning.
  2. The Large Language Model ($L$): A massive model (such as T0pp) with extensive general knowledge but limited ability to adhere to strict classification constraints in a zero-shot setting.

The Workflow

The Primed+ system operates through a three-step pipeline, mathematically represented as $E(L(E(X)))$:

  • Initial Inference: The Entailment Predictor ($E$) makes an initial prediction based on the input text ($X$).
  • Knowledge Priming: This initial guess is fed into the LLM ($L$) via a prompt (e.g., "Here is a text that entails [Guess]. What area is this related to?"). The LLM uses its internal knowledge to expand upon the guess, identifying parent categories or providing descriptive context.
  • Final Classification: This expanded, "primed" information is fed back into the Entailment Predictor to make the final, more accurate classification.

Experimental Setup and Datasets

To test the framework's efficacy in long-tail scenarios, the authors refactored two popular datasets to mimic real-world hierarchical distributions:

  • Web of Science (WOS): Academic abstracts spanning various disciplines.
  • Amazon Beauty: Product reviews with deep hierarchical categories (e.g., Beauty > Skin Care > Face > Creams & Moisturizers).

The researchers created versions of these datasets with varying tree depths (up to 5 levels) to evaluate how models handle increasingly specific leaf-node predictions.

Key Results and Insights

The Primed+ method consistently outperformed standalone models across all tested metrics, particularly in Macro F1 scores, which are sensitive to long-tail performance.

  • Amazon Beauty (Depth 2): A standalone T0pp (LLM) achieved a Macro F1 of only 6.01. The baseline Entailment model reached 51.15. The Primed+ framework improved this to 52.84.
  • Web of Science: The framework achieved a 69.48 Macro F1, significantly outperforming the standalone LLM (11.04) and the baseline Entailment model (68.93).
  • Convergence: Top-N analysis revealed that the Primed+ model converges to high accuracy much faster than baselines, indicating that its primary guesses are significantly more reliable.

LLMs as Reasoning Bridges

A central insight of this research is that LLMs are most effective when used as "reasoning bridges" rather than final decision-makers. By helping a specialized entailment model understand the relationship between a text and a rare label, the LLM leverages its strengths without being hampered by its weaknesses in strict classification. Furthermore, because the method is parameter-free, it offers a cost-effective solution for industrial applications where fine-tuning large models is impractical.

Limitations and Future Directions

While effective, the framework has areas for refinement. The authors noted that incorporating "contradiction logic" (identifying what a text definitely does not represent) currently introduces noise into the prompts. Additionally, while the function $E(L(E(X)))$ could theoretically be applied recursively, the study observed diminishing returns beyond the initial priming. Finally, like many LLM-based approaches, the system remains sensitive to prompt engineering, requiring careful selection to optimize performance across different domains.

Summary updated

BibTeX
@inproceedings{bhambhoria2023simple,
  author = {Rohan Bhambhoria and Lei Chen and Xiaodan Zhu},
  title = {{A Simple and Effective Framework for Strict Zero-Shot Hierarchical Classification}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
  doi = {10.18653/v1/2023.acl-short.152},
}
Figure