In standard image classification, we treat labels as a "flat" list of independent categories. To a typical neural network, the semantic distance between a "Husky" and a "Beagle" is exactly the same as the distance between a "Husky" and a "Toaster." This approach ignores the rich, structured nature of human knowledge, where objects exist within nested taxonomies—a Husky is a Dog, which is a Mammal, which is an Animal.
Research into taxonomy-aware learning seeks to move beyond this flat architecture by baking hierarchical relationships directly into the model's structure and loss functions. By doing so, researchers aim to make models more robust, more efficient in data-scarce scenarios, and more "semantically reasonable" in their mistakes.
The Geometry of Hierarchy
One of the most intuitive ways to incorporate a taxonomy is to change the geometry of the model's internal representation. If we know that classes follow a tree structure, we can force the model's embedding space to reflect that tree.
Baoquan Zhang and Rui Ye explored this using hyperbolic space. Unlike standard Euclidean space, hyperbolic geometry naturally expands exponentially, making it a perfect fit for tree-like structures where the number of nodes increases as you move from the root (coarse) to the leaves (fine). By embedding taxonomies like WordNet into hyperbolic space, their HyperKT framework allows models to transfer knowledge from common parent classes to rare, fine-grained child classes more effectively.
Other researchers, such as those in Connecting Sphere Manifolds Hierarchically for Regularization, use nested spherical constraints. In this setup, the classifier for a specific species is forced to lie on a manifold centered around its genus, which in turn is centered around its family. This recursive nesting acts as a powerful structural prior, guiding the model toward a more logical organization of features.
Propagating Information Across Scales
Beyond static embeddings, many frameworks focus on how information flows between different levels of granularity during the inference process. The goal is to ensure that the "coarse" understanding of an object (it's a bird) informs the "fine" identification (it's a Laysan Albatross).
Greg Mori and Nelson Nauata introduced the Bidirectional Inference Neural Network (BINN) to handle this. Instead of a single prediction, the BINN propagates signals up and down the label hierarchy, allowing the model to reconcile potentially conflicting predictions across different levels of abstraction. Similarly, Yanming Guo and colleagues proposed a CNN-RNN framework that treats hierarchical classification as a sequence-to-sequence problem, where the model predicts labels from the most general to the most specific.
This "coarse-to-fine" progression is also used as a curriculum for difficult tasks like domain adaptation. Sinan Wang and Yunbo Wang developed Progressive Adversarial Networks (PAN), which align domains by first matching easy, coarse categories before attempting to align the subtle differences required for fine-grained recognition.
Enforcing Logical and Visual Consistency
A common failure mode in hierarchical systems is "semantic incoherence"—a model might correctly identify an object as a "Cat" at a coarse level but then misclassify it as a "Beagle" at a fine level.
To solve this, Seulki Park and Stella X. Yu introduced H-CAST, which enforces visual consistency. They argue that if a model identifies a fine-grained part (like a wing), that feature must be spatially contained within the coarse-grained object (the bird). By aligning the model's attention across levels, they ensure that the hierarchy is not just a label constraint but a reflection of the physical world.
For models already trained, Kanishk Jain and Vineet Gandhi proposed Hierarchical Ensembles (HiE). This is a post-hoc "sanity check" where a coarse classifier amends the fine-grained prediction at test time. If the fine-grained model predicts a species that doesn't belong to the coarse model's predicted genus, the system corrects the error, significantly reducing the "severity" of mistakes.
The Bottleneck of Predefined Taxonomies
While these methods show great promise, they share a common Achilles' heel: a heavy reliance on predefined, static taxonomies. Most current research, including work by Aoxue Li and Liwei Wang, assumes that an expert-curated tree (like WordNet or a biological phylogeny) is available.
This dependency creates several problems:
- Rigidity: Human-defined taxonomies may not align with the visual features a model actually sees. A "coat" and a "jacket" might be in different taxonomic branches but look nearly identical.
- Incompleteness: In niche industrial domains or abstract datasets, no formal hierarchy may exist.
- Granularity Competition: As noted by Zhiguang Lu and Qianqian Xu, coarse labels are often easier to learn and can "dominate" the feature encoder, preventing the model from learning the subtle details needed for fine-grained tasks.
To address this, recent work by Hazel Doughty and Cees Snoek suggests moving toward Generalized Category Discovery, where models must learn to categorize new objects without a pre-existing map. Similarly, Alexandre Sablayrolles and Matthijs Douze with Grafit have begun exploring how to learn fine-grained representations using only coarse labels, effectively asking the model to discover the sub-hierarchy on its own.