Abstract
Hierarchical Text Classification (HTC) is a challenging task that organizes texts into categories arranged in a hierarchy. Zero-Shot Hierarchical Text Classification (ZS-HTC) further assumes that only the hierarchical taxonomy is available, without any training data. Existing ZS-HTC methods typically rely on prototype-based frameworks that embed category names into vectors. However, these often underperform due to the ambiguity and impreciseness of category names. In this paper, we propose HierPrompt, a method that leverages hierarchy-aware prompts to instruct Large Language Models (LLMs) to produce more representative and informative prototypes. Specifically, we introduce the Example Text Prototype (ETP) alongside the Category Name Prototype (CNP) to enrich hierarchical information. We also propose a Maximum Similarity Propagation (MSP) technique to incorporate the hierarchy into similarity calculations. Experiments on three benchmark datasets demonstrate that HierPrompt substantially outperforms existing ZS-HTC methods.
Introduction: The Challenge of Zero-Shot Classification
Hierarchical Text Classification involves organizing documents into tree-like structures (e.g., Sports → Basketball → NBA). In the zero-shot setting, models must perform this task without labeled training data, relying solely on category names and their positions in the hierarchy.
Existing methods typically use a prototype-based approach, turning category names into vector embeddings. However, this approach faces two critical hurdles:
- Ambiguity: Category names like "Work" are often too vague. Without context, it is unclear if the term refers to employment, creative works, or physical labor.
- Stylistic Mismatch: A label like "Video Games" does not capture the specific style of the target text. A Wikipedia entry for a game differs significantly in tone and structure from a casual customer review on Amazon.
The HierPrompt Framework
HierPrompt addresses these issues by using the reasoning power of LLMs to "flesh out" prototypes through three main components: Category Name Contextualization, Example Text Generation, and Maximum Similarity Propagation.
1. Category Name Prototypes (CNP)
Instead of embedding raw category names, HierPrompt uses hierarchy-aware prompts. It instructs an LLM to summarize a category by considering its parents, siblings, and children. For instance, if "Work" has children like "Song" and "Software," the LLM identifies the category as "Creative Works" and generates a descriptive sentence, leading to a much more accurate embedding.
2. Example Text Prototypes (ETP)
To bridge the stylistic gap, the authors introduce the Chain of Generation (COG). This process prompts the LLM to:
- Understand the category semantics.
- Exemplify specific instances (e.g., "Think of a specific basketball player").
- Generate a synthetic document in the style of the target dataset (e.g., a fake Amazon review). These synthetic examples are embedded and averaged to create a prototype that mimics the actual data's distribution.
3. Maximum Similarity Propagation (MSP)
Standard models often treat hierarchy levels independently. HierPrompt introduces MSP, allowing parent categories to "absorb" the maximum similarity score from their descendants. This follows the logic of transitivity: if a text is a perfect match for "NBA," it is inherently a strong match for its parent category, "Basketball."
Experimental Evaluation
The researchers evaluated HierPrompt on three widely used benchmarks: NYT (formal news), DBpedia (Wikipedia excerpts), and Amazon (informal product reviews).
Key Results and Insights
- Significant Performance Gains: On the DBpedia dataset, HierPrompt outperformed the previous state-of-the-art (HiLA) by 5.41%, 5.24%, and 4.99% across levels 1, 2, and 3, respectively.
- Disambiguating Coarse Labels: Classification at the top level (L1) is traditionally the most difficult due to label ambiguity. HierPrompt’s contextualization provided the most dramatic gains here; on DBpedia L1, the Macro-F1 score improved from 31.60% (base) to 76.95%.
- Style Sensitivity: Generating synthetic data that mimics the target domain's "vibe" proved more effective than simple dictionary definitions, highlighting the importance of stylistic alignment in zero-shot tasks.
Conclusion
HierPrompt demonstrates that LLMs can effectively disambiguate hierarchical taxonomies and generate style-accurate synthetic data to improve zero-shot classification. While the method is robust across formal and informal styles, future work will address potential LLM hallucinations in niche taxonomies and explore joint optimization of textual descriptions and the underlying embedding space.
Summary updated