TransHP: Image Classification with Hierarchical Prompting

Wenhao Wang, Yifan Sun, Wei Li, Yi Yang

Abstract

This paper explores a hierarchical prompting mechanism for the hierarchical image classification (HIC) task. Different from prior HIC methods, our hierarchical prompting is the first to explicitly inject ancestor-class information as a tokenized hint that benefits the descendant-class discrimination. We model this prompting mechanism into a Transformer with Hierarchical Prompting (TransHP). TransHP consists of three steps: 1) learning a set of prompt tokens to represent the coarse (ancestor) classes, 2) on-the-fly predicting the coarse class of the input image at an intermediate block, and 3) injecting the prompt token of the predicted coarse class into the intermediate feature. TransHP improves image classification accuracy (e.g., +2.83% on ImageNet), training data efficiency (e.g., +12.69% improvement under 10% ImageNet training data), and model explainability.

Introduction

Standard image classification models typically treat labels as a "flat" list, ignoring the inherent semantic relationships between objects. In reality, categories are hierarchical: a Goldfish is a type of Fish, which is a type of Vertebrate. While Hierarchical Image Classification (HIC) aims to exploit these relationships, prior methods often rely on complex loss functions or specialized embedding spaces that do not explicitly guide the model's attention during inference. TransHP mimics human visual recognition by identifying a broad category first to narrow the scope, then shifting focus to subtle variations within that category.

Methods

The TransHP architecture modifies a standard Vision Transformer (ViT) through a three-step hierarchical prompting process:

  1. Learning Prompt Tokens: The model maintains a "prompt pool" of learnable tokens representing coarse (ancestor) classes.
  2. On-the-fly Prediction: At an intermediate "prompting block," the model predicts the coarse class of the input image based on current features.
  3. Prompt Injection: The predicted coarse-class prompt token is injected into the feature stream. Through self-attention, image tokens "absorb" this information, conditioning subsequent feature extraction.
Comparison of (a) baseline ViT and (b) TransHP architecture showing the hierarchical prompting mechanism and resulting attention map shift.

This mechanism encourages the model to focus on specific, discriminative regions (like a bird's beak) rather than the entire foreground, effectively using the coarse category as a hint to refine its search.

Results and Key Insights

TransHP was evaluated across five diverse datasets, including ImageNet-1000, iNaturalist, and CIFAR-100.

  • Accuracy Boost: On ImageNet, TransHP improved the ViT-B/16 baseline by +2.83% in top-1 accuracy. It consistently outperformed recent state-of-the-art HIC methods like Guided and HiMulConE.
  • Low-Data Dominance: When trained on only 10% of ImageNet data, TransHP outperformed the baseline by a staggering +12.69%. This suggests that hierarchical structure provides a powerful inductive bias when data is scarce.
  • Explainability: Visualizations show that while a baseline ViT attends to the entire body of an animal, TransHP—once prompted—shifts its attention to highly specific, informative regions (e.g., the face and crown of a goldfish).
  • Minimal Overhead: The approach adds only ~0.6 million parameters to a ViT-B/16 backbone (a ~2.7% increase) while providing significant performance gains.

Conclusion

TransHP demonstrates that explicitly injecting hierarchical information as prompts significantly benefits fine-grained discrimination. By mimicking the human "coarse-to-fine" recognition process, the model achieves superior accuracy and data efficiency with minimal architectural overhead. Future work may extend this mechanism to other tasks like semantic segmentation or explore automated schemes for optimal prompting block placement.

Summary updated

Abstract

This paper explores a hierarchical prompting mechanism for the hierarchical image classification (HIC) task. Different from prior HIC methods, our hierarchical prompting is the first to explicitly inject ancestor-class information as a tokenized hint that benefits the descendant-class discrimination. We think it well imitates human visual recognition, i.e., humans may use the ancestor class as a prompt to draw focus on the subtle differences among descendant classes. We model this prompting mechanism into a Transformer with Hierarchical Prompting (TransHP). TransHP consists of three steps: 1) learning a set of prompt tokens to represent the coarse (ancestor) classes, 2) on-the-fly predicting the coarse class of the input image at an intermediate block, and 3) injecting the prompt token of the predicted coarse class into the intermediate feature. Though the parameters of TransHP maintain the same for all input images, the injected coarse-class prompt conditions (modifies) the subsequent feature extraction and encourages a dynamic focus on relatively subtle differences among the descendant classes. Extensive experiments show that TransHP improves image classification on accuracy (e.g., improving ViT-B/16 by +2.83% ImageNet classification accuracy), training data efficiency (e.g., +12.69% improvement under 10% ImageNet training data), and model explainability. Moreover, TransHP also performs favorably against prior HIC methods, showing that TransHP well exploits the hierarchical information.
BibTeX
@inproceedings{wang2023transhp,
  author = {Wenhao Wang and Yifan Sun and Wei Li and Yi Yang},
  title = {{TransHP: Image Classification with Hierarchical Prompting}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2023},
  url = {https://openreview.net/forum?id=vpQuCsZXz2},
}
Figure