HierPrompt: Zero-Shot Hierarchical Text Classification with LLM-Enhanced Prototypes

Qian Zhang, Qinliang Su, Wei Zhu, Pang YachunEMNLP 2025 · OpenReview

Abstract

Hierarchical Text Classification (HTC) is a challenging task that organizes texts into categories arranged in a hierarchy. Zero-Shot Hierarchical Text Classification (ZS-HTC) further assumes that only the hierarchical taxonomy is available, without any training data. Existing ZS-HTC methods typically rely on prototype-based frameworks that embed category names into vectors. However, these often underperform due to the ambiguity and impreciseness of category names. In this paper, we propose HierPrompt, a method that leverages hierarchy-aware prompts to instruct Large Language Models (LLMs) to produce more representative and informative prototypes. Specifically, we introduce the Example Text Prototype (ETP) alongside the Category Name Prototype (CNP) to enrich hierarchical information. We also propose a Maximum Similarity Propagation (MSP) technique to incorporate the hierarchy into similarity calculations. Experiments on three benchmark datasets demonstrate that HierPrompt substantially outperforms existing ZS-HTC methods.

Introduction: The Challenge of Zero-Shot Classification

Hierarchical Text Classification involves organizing documents into tree-like structures (e.g., Sports → Basketball → NBA). In the zero-shot setting, models must perform this task without labeled training data, relying solely on category names and their positions in the hierarchy.

Existing methods typically use a prototype-based approach, turning category names into vector embeddings. However, this approach faces two critical hurdles:

  1. Ambiguity: Category names like "Work" are often too vague. Without context, it is unclear if the term refers to employment, creative works, or physical labor.
  2. Stylistic Mismatch: A label like "Video Games" does not capture the specific style of the target text. A Wikipedia entry for a game differs significantly in tone and structure from a casual customer review on Amazon.
Figure 1: Ambiguity and stylistic mismatch in category names across different datasets (DBpedia vs. Amazon).

The HierPrompt Framework

HierPrompt addresses these issues by using the reasoning power of LLMs to "flesh out" prototypes through three main components: Category Name Contextualization, Example Text Generation, and Maximum Similarity Propagation.

Figure 2: Overview of the HierPrompt framework, illustrating Category Name Contextualization and Example Text Generation via Chain of Generation (COG).

1. Category Name Prototypes (CNP)

Instead of embedding raw category names, HierPrompt uses hierarchy-aware prompts. It instructs an LLM to summarize a category by considering its parents, siblings, and children. For instance, if "Work" has children like "Song" and "Software," the LLM identifies the category as "Creative Works" and generates a descriptive sentence, leading to a much more accurate embedding.

2. Example Text Prototypes (ETP)

To bridge the stylistic gap, the authors introduce the Chain of Generation (COG). This process prompts the LLM to:

  • Understand the category semantics.
  • Exemplify specific instances (e.g., "Think of a specific basketball player").
  • Generate a synthetic document in the style of the target dataset (e.g., a fake Amazon review). These synthetic examples are embedded and averaged to create a prototype that mimics the actual data's distribution.

3. Maximum Similarity Propagation (MSP)

Standard models often treat hierarchy levels independently. HierPrompt introduces MSP, allowing parent categories to "absorb" the maximum similarity score from their descendants. This follows the logic of transitivity: if a text is a perfect match for "NBA," it is inherently a strong match for its parent category, "Basketball."

Experimental Evaluation

The researchers evaluated HierPrompt on three widely used benchmarks: NYT (formal news), DBpedia (Wikipedia excerpts), and Amazon (informal product reviews).

Key Results and Insights

  • Significant Performance Gains: On the DBpedia dataset, HierPrompt outperformed the previous state-of-the-art (HiLA) by 5.41%, 5.24%, and 4.99% across levels 1, 2, and 3, respectively.
  • Disambiguating Coarse Labels: Classification at the top level (L1) is traditionally the most difficult due to label ambiguity. HierPrompt’s contextualization provided the most dramatic gains here; on DBpedia L1, the Macro-F1 score improved from 31.60% (base) to 76.95%.
  • Style Sensitivity: Generating synthetic data that mimics the target domain's "vibe" proved more effective than simple dictionary definitions, highlighting the importance of stylistic alignment in zero-shot tasks.

Conclusion

HierPrompt demonstrates that LLMs can effectively disambiguate hierarchical taxonomies and generate style-accurate synthetic data to improve zero-shot classification. While the method is robust across formal and informal styles, future work will address potential LLM hallucinations in niche taxonomies and explore joint optimization of textual descriptions and the underlying embedding space.

Summary updated

Abstract

Hierarchical Text Classification is a challenging task which classifies texts into categories arranged in a hierarchy. Zero‐Shot Hierarchical Text Classification (ZS-HTC) further assumes only the availability of hierarchical taxonomy, without any training data. Existing works of ZS-HTC are typically built on the prototype-based framework by embedding the category names into prototypes, which, however, do not perform very well due to the ambiguity and impreciseness of category names. In this paper, we propose HierPrompt, a method that leverages hierarchy-aware prompts to instruct LLM to produce more representative and informative prototypes. Specifically, we first introduce Example Text Prototype (ETP), in conjunction with Category Name Prototype (CNP), to enrich the information contained in hierarchical prototypes. A Maximum Similarity Propagation (MSP) technique is also proposed to consider the hierarchy in similarity calculation. Then, the hierarchical prototype refinement module is utilized to (i) contextualize the category names for more accurate CNPs and (ii) produce detailed example texts for each leaf category to form ETPs. Experiments on three benchmark datasets demonstrate that HierPrompt substantially outperforms existing ZS‐HTC methods.
BibTeX
@inproceedings{zhang2025hierprompt,
  author = {Qian Zhang and Qinliang Su and Wei Zhu and Pang Yachun},
  title = {{HierPrompt: Zero-Shot Hierarchical Text Classification with LLM-Enhanced Prototypes}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
  url = {https://openreview.net/forum?id=BEIpaRWwOp},
}
Figure