Introduction
Vision-language models (VLMs) like CLIP have demonstrated remarkable zero-shot capabilities in identifying fine-grained object categories. However, these models often exhibit a fundamental lack of semantic understanding regarding the hierarchical relationships between classes. For instance, a model might correctly identify a "tiger" at the leaf level but simultaneously classify it as a "person" at a higher taxonomic level. This phenomenon, termed hierarchical inconsistency, limits the reliability of VLMs in real-world applications requiring multi-granularity classification.
To address this, we present ProTeCt (Prompt Tuning for Hierarchical Consistency), a novel strategy designed to enforce logical "is-a" relationships during the prompt-tuning phase. ProTeCt ensures that models remain consistent across all levels of semantic granularity without sacrificing fine-grained accuracy.
The Problem: Hierarchical Inconsistency
Standard training paradigms for image classification typically focus on "leaf" labels—the most specific categories available (e.g., "Golden Retriever" or "Fox Squirrel"). While effective for flat classification, this approach ignores the inherent structure of the world. In Taxonomic Open Set (TOS) classification, a model must be able to provide labels at varying degrees of abstraction depending on the user's needs.
Current state-of-the-art models adapted via prompt tuning (like CoOp or MaPLe) often produce nonsensical results across a taxonomy. They may correctly identify a species but fail to recognize its correct genus or family. This occurs because standard cross-entropy loss only penalizes the final label prediction, leaving the internal nodes of the hierarchy unconstrained.
The ProTeCt Framework
ProTeCt is a "plug-and-play" method that can be integrated into existing prompt-tuning techniques. It introduces two primary components to enforce hierarchical logic:
1. Node-Centric Loss (NCL)
NCL ensures that for every internal node in a taxonomy (e.g., "Mammal"), the model can correctly distinguish between its immediate children (e.g., "Dog," "Cat," and "Tiger"). By supervising these local decision points, the model learns the relative differences between sibling categories at every level of the tree.
2. Dynamic Treecut Loss (DTL)
The core innovation of ProTeCt is the Treecut Sampler. A taxonomic tree can be "cut" in billions of ways to create valid, mutually exclusive label sets at different levels of abstraction. During training, the Treecut Sampler randomly selects these different "levels," forcing the model to learn how to classify images across varying degrees of abstraction simultaneously.
By combining NCL and DTL, the model learns that a "tiger" is not just an isolated label, but a member of a nested set of categories: "feline," "mammal," and "physical entity."
Experimental Evaluation
The researchers evaluated ProTeCt across several standard benchmarks and their hierarchical extensions:
- CIFAR-100: Small-image dataset with a 20-superclass hierarchy.
- SUN: Scene recognition dataset with a shallow hierarchy.
- ImageNet: The primary benchmark, utilizing the deep WordNet hierarchy.
- Generalization Variants: To assess robustness, tests were conducted on ImageNet-v2, ImageNet-Sketch, ImageNet-A (Adversarial), and ImageNet-R (Renditions).
Results and Discussion
The quantitative improvements provided by ProTeCt are significant, particularly in metrics that account for hierarchical structure.
Hierarchical Consistent Accuracy (HCA)
HCA measures whether a model correctly identifies the leaf node and all its ancestors. On ImageNet, using the CoOp method, HCA improved from a baseline of 2.99% to 37.74% after applying ProTeCt. This massive jump indicates that while vanilla models often "guess" the right leaf label, ProTeCt-tuned models actually understand the path to that label.
Mean Treecut Accuracy (MTA)
MTA evaluates performance across random levels of the tree, simulating real-world TOS queries. Performance improved from 46.98% to 88.61%, demonstrating that the model is far more robust to queries at different levels of abstraction.
No Accuracy Trade-off
Critically, ProTeCt achieves these consistency gains without degrading "leaf" accuracy. In many cases, the leaf accuracy remained identical to or slightly exceeded that of vanilla prompt-tuning methods. This suggests that hierarchical constraints act as a beneficial regularizer rather than a restrictive bottleneck.
Conclusion
ProTeCt addresses a critical blind spot in current vision-language models: the lack of hierarchical awareness. By introducing node-centric supervision and dynamic treecut sampling, ProTeCt transforms prompt tuning from a simple label-matching exercise into a process that respects semantic structure. This leads to models that are not only accurate at the fine-grained level but also logically consistent and robust across the entire taxonomy.
Summary updated