In standard machine learning, we typically treat classification labels as an independent, flat set. If a model is trained on a long-tailed dataset—where a few "head" classes have thousands of examples and many "tail" classes have only a handful—this flat assumption causes severe performance degradation. The model treats a rare class like Ocelot completely separately from a common class like Cat, failing to realize that they share a massive amount of underlying structural, visual, and semantic information.
Hierarchical feature sharing bypasses data scarcity by forcing rare categories to inherit or borrow features from their data-rich ancestors or neighbors within a taxonomy. Instead of forcing a tail class to learn representations from scratch, the model leverages the stable, well-sampled feature spaces of head classes to regularize and anchor the tail.
Top-Down Inheritance and Parameter Tying
The most direct way to share features is through a structured taxonomy where fine-grained tail classes are explicitly modeled as children of coarse-grained head classes. A rare sub-category does not need to learn fundamental features (such as basic visual shapes or linguistic contexts) because those are already captured by its parent category.
To operationalize this, Han et al. incorporate taxonomic hierarchies as structural priors during training. By tying the parameters of rare classes to their common ancestors, the architecture creates a strong regularization effect that prevents the model from overfitting to the few available tail samples. Similarly, Yu et al. employ a top-down classification strategy where fine-grained relations inherit representational strength from broader parent categories.
This strategy builds on foundational insights from early systems like the one described in Modeling Commonality among Related Classes in Relation Extraction. The core intuition is that by learning a generalized discriminative function at a high level of abstraction, the model only needs to use its scarce tail data to learn the specific, fine-grained variations that distinguish a subtype from its parent.
Prototype Propagation on Category Graphs
Rather than relying on rigid parameter tying, modern architectures frequently model taxonomies as graphs, using message-passing and attention mechanisms to propagate knowledge dynamically across nodes.
For instance, Liu et al. developed Prototype Propagation Networks (PPN), which use attention mechanisms to pass semantic information from data-rich parent prototypes to data-scarce child prototypes. This allows rare leaf nodes to inherit a robust semantic prior before processing instance-level data. In multi-label settings, Rios et al. combine label-wise attention with Graph Convolutional Networks (GCNs) to smooth the label space, enabling effective knowledge transfer even to completely unseen zero-shot categories.
This graph-based smoothing can also be achieved by mapping labels and features into a shared space. As demonstrated in the work of Shen, embedding hierarchical label structures into the same latent space as input features allows models to generalize to tail labels based on their structural proximity to well-sampled parent nodes. Even when explicit graphs are absent, structural knowledge can be retained via memory systems; Binhua Li utilizes a memory matrix learned from a broad set of base classes to dynamically refine the representations of novel, data-scarce categories through dynamic routing.
Borrowing from the Rich via Latent Taxonomies
What happens if a clean, human-annotated taxonomy does not exist? The underlying intuition of hierarchical sharing still applies: models can discover latent groupings or directly borrow features from semantically similar head classes.
To address datasets lacking explicit hierarchies, Du et al. automatically discover a hierarchy of "super-classes" during training, allowing tail classes to borrow discriminative power from head classes across a learned graph. Alternatively, features can be compositionally transferred; Parisot et al. show that models can recognize rare categories by recomposing the learned high-level features of semantically similar, well-represented classes.
In computer vision tasks like scene graph generation, where relationship predicates are highly imbalanced, Gao et al. mitigate data starvation by hallucinating features for tail relations. They achieve this by combining the original visual signal with representative semantic vectors borrowed directly from common head relations, ensuring the tail representations remain structurally robust.
Multi-Level Supervision as an Architectural Regularizer
Finally, forcing a model to simultaneously predict multiple levels of a taxonomy acts as a powerful regularizer for the underlying feature extractor. When a network is tasked with classifying an instance at the root, coarse, and fine-grained levels at the same time, the shared backbone is forced to prioritize features that remain invariant across scales.
For example, Peri demonstrated that multi-level taxonomic supervision significantly improves 3D object detection for rare classes by forcing collaborative feature sharing across the entire dataset. This architectural constraint ensures that even if a model rarely encounters a specific tail class, its feature extractors are highly tuned to the broader, data-abundant parent category. The model maintains a stable feature space, ensuring that tail performance is supported by the collective experience of the entire taxonomic tree.