TransHP: Image Classification with Hierarchical Prompting

Wenhao Wang, Yifan Sun, Wei Li, Yi Yang

TL;DR

TransHP introduces a "hierarchical prompting" mechanism for Vision Transformers that mimics human visual recognition. By injecting coarse-category hints (e.g., identifying an object as a "bird" first) into intermediate layers, the model guides subsequent feature extraction. This approach achieves significantly higher accuracy and remarkable data efficiency in fine-grained classification tasks.

The Challenge of Hierarchical Image Classification

In standard image classification, models typically map an input image directly to a specific label (like "Goldfish") in a single pass. However, many datasets possess an inherent hierarchy: a "Goldfish" is a type of "Fish," which is an "Animal."

While Hierarchical Image Classification (HIC) aims to exploit these relationships, previous methods often treated coarse labels merely as additional supervision via extra loss functions. They failed to use this coarse information to dynamically change how the model processes the image. Humans, on the other hand, recognize objects hierarchically—we identify the general category first, which then prompts us to look for specific, subtle details that distinguish species within that category. TransHP aims to bake this "zoom-in" logic directly into the Transformer architecture.

Introducing TransHP

The authors propose TransHP (Transformer with Hierarchical Prompting), which modifies a standard Vision Transformer (ViT) to incorporate hierarchical context dynamically.

Comparison of standard ViT and TransHP architectures

The architecture introduces a "prompting block" and operates in three key steps:

  1. Learning a Prompt Pool: The model maintains a set of learnable "prompt tokens." Each token represents a coarse ancestor class (e.g., one token for "mammals," one for "birds").
  2. On-the-fly Prediction: At a specific intermediate layer (the prompting block), the model uses the current image features to predict which coarse class the image belongs to.
  3. Prompt Injection: The model selects the corresponding prompt token from the pool and injects it into the feature stream. Through cross-attention, the existing image tokens "absorb" information from this prompt.
Detailed view of the prompting block and coarse-class prediction

Crucially, this process is dynamic. While the model's parameters are fixed after training, the specific prompt injected changes based on the input image. This conditions the remaining layers of the Transformer to focus on the subtle variations necessary to distinguish between closely related descendant classes. Unlike traditional prompting that happens at the input layer, TransHP's intermediate injection allows the bottom layers to learn shared, general features while the top layers become specialized experts for specific categories.

Key Insights and Results

The researchers evaluated TransHP across diverse datasets with hierarchical structures, including ImageNet-1000, iNaturalist, CIFAR-100, and DeepFashion-inshop.

  • The "Aha!" Moment in Attention: Visualizations reveal a striking shift in the model's focus. Before the prompt injection, the model's attention is spread across the entire object. After receiving a coarse prompt (e.g., "fish"), the attention in the final layers shifts specifically to discriminative features like the head or fins.
  • Autonomous Selection: The model isn't "told" the coarse label at inference time. It learns to predict the coarse category and select the right prompt autonomously. As training progresses, the model's "absorption weights" naturally gravitate toward the correct target prompt.
  • Accuracy Boost: TransHP improved the top-1 accuracy of a ViT-B/16 on ImageNet by +2.83%, consistently outperforming other state-of-the-art HIC methods like Guided and HiMulConE.
  • Extreme Data Efficiency: When trained on only 10% of ImageNet data, TransHP outperformed the baseline by a massive +12.69%. Establishing a "coarse" foundation allows the model to learn fine-grained details much faster with fewer examples.

Limitations and Future Work

  • Prompting Block Placement: There is currently no mathematical formula for the "perfect" layer to insert the prompts. The optimal position depends on the dataset's hierarchy (e.g., if there are many coarse classes, the block should be higher up), requiring empirical tuning.
  • Computational Overhead: While inference is light, training time increases by about 12-15% due to the extra loss calculations for the intermediate coarse predictions.
  • Task Expansion: The study focuses strictly on image classification, but the authors note that other tasks, such as semantic segmentation, could benefit from hierarchical annotations in future research.

Summary updated

Abstract

This paper explores a hierarchical prompting mechanism for the hierarchical image classification (HIC) task. Different from prior HIC methods, our hierarchical prompting is the first to explicitly inject ancestor-class information as a tokenized hint that benefits the descendant-class discrimination. We think it well imitates human visual recognition, i.e., humans may use the ancestor class as a prompt to draw focus on the subtle differences among descendant classes. We model this prompting mechanism into a Transformer with Hierarchical Prompting (TransHP). TransHP consists of three steps: 1) learning a set of prompt tokens to represent the coarse (ancestor) classes, 2) on-the-fly predicting the coarse class of the input image at an intermediate block, and 3) injecting the prompt token of the predicted coarse class into the intermediate feature. Though the parameters of TransHP maintain the same for all input images, the injected coarse-class prompt conditions (modifies) the subsequent feature extraction and encourages a dynamic focus on relatively subtle differences among the descendant classes. Extensive experiments show that TransHP improves image classification on accuracy (e.g., improving ViT-B/16 by +2.83% ImageNet classification accuracy), training data efficiency (e.g., +12.69% improvement under 10% ImageNet training data), and model explainability. Moreover, TransHP also performs favorably against prior HIC methods, showing that TransHP well exploits the hierarchical information. The code is available at: https://github.com/WangWenhao0716/TransHP.
BibTeX
@inproceedings{wang2023transhp,
  author = {Wenhao Wang and Yifan Sun and Wei Li and Yi Yang},
  title = {{TransHP: Image Classification with Hierarchical Prompting}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2023},
  url = {https://openreview.net/forum?id=vpQuCsZXz2},
}
Figure