Hypernym Bias: Unraveling Deep Classifier Training Dynamics through the Lens of Class Hierarchy

Roman Malashin, Valeria Yachnaya, Alexandr V. MullinAISTATS 2025

Introduction

Deep neural networks do not learn all classes at a uniform rate. Instead, they exhibit a phenomenon termed "Hypernym Bias," where they master broad, high-level categories (e.g., "animals") significantly earlier than they distinguish specific sub-categories (e.g., "tabby cats"). This research suggests that the training process can be interpreted as a top-to-bottom hierarchical clustering of labels, bridging the gap between early-stage "simplicity bias" and the late-stage phenomenon known as "Neural Collapse."

Methodology

The study introduces a framework to track the evolution of the feature manifold using several key techniques:

  1. Greedy Hypernym Classification: Mapping ImageNet classes to WordNet superclasses to compare accuracy across hierarchy levels.
  2. Isomorphic Random Baselines: Using random groupings of identical size to control for the statistical ease of classifying fewer categories.
  3. Manifold Alignment: Utilizing the Cophenetic Correlation Coefficient (CCC) to measure the alignment between internal feature distances and semantic WordNet distances.
  4. Hypernym Neural Collapse: Adapting standard collapse metrics to observe geometric simplification at different hierarchical levels.

The Hypernym Bias Phenomenon

The core finding is that networks learn the "big picture" relationships before fine-grained details. This is not a quirk of specific models but a fundamental property of gradient-based learning across diverse architectures.

Relative accuracy and feature manifold evolution across architectures.

As shown in the figure above, across architectures like ResNet, MobileNet, and ViT, the relative accuracy for hypernym superclasses (orange dashed line) rises much faster than for random superclasses (grey dash-dotted line) or the base hyponyms (blue solid line). For instance, a ResNet-50 reaches 95% relative accuracy for top-level hypernyms in just 6 epochs, compared to 64 epochs for random groupings. The manifold visualizations (part b) further illustrate how the feature space progressively clusters from a random initialization into distinct semantic regions over hundreds of epochs.

Hierarchical Neural Collapse

Neural Collapse typically refers to the terminal phase of training where within-class variance drops to zero and class centers form a rigid geometric structure. This research demonstrates that this collapse happens progressively through the hierarchy.

Neural Collapse metrics showing earlier convergence for hypernym superclasses.

The metrics for within-class variation and nearest class-center simplification (shown above) reveal that the properties of Neural Collapse manifest in the hypernym label space much earlier than in the final label space. This suggests that the network stabilizes high-level features first, likely because neurons detecting features shared across a hypernym (e.g., "fur" for animals) are activated more frequently and thus trained faster than those required for fine-grained distinction.

Conclusion and Future Work

Hypernym Bias provides a new lens for understanding training dynamics, viewing the process as a hierarchical carving of the feature space. While the current work relies on external hierarchies like WordNet, future research aims to discover these hierarchies directly from data and explore whether incorporating hierarchical information into loss functions can enhance training efficiency or robustness.

Summary updated

BibTeX
@inproceedings{malashin2025hypernym,
  author = {Roman Malashin and Valeria Yachnaya and Alexandr V. Mullin},
  title = {{Hypernym Bias: Unraveling Deep Classifier Training Dynamics through the Lens of Class Hierarchy}},
  booktitle = {International Conference on Artificial Intelligence and Statistics},
  year = {2025},
}
Figure