Learning Taxonomic Trees with Hierarchical Representation Regularization for Large Multimodal Models

Hulingxiao He, Zhi Tan, Yuxin Peng

TL;DR

Large multimodal models (LMMs) struggle with hierarchical visual recognition (HVR) because they lack explicit taxonomic knowledge. This paper introduces Hierarchical Representation Regularization (HiR²), a plug-and-play method that injects taxonomic structure into LMMs' internal representations using hyperbolic geometry and spherical angular separation, significantly improving HVR consistency.

What Problem Does This Paper Address?

Modern large multimodal models (LMMs) excel at many vision-language tasks, but they often fail to grasp the hierarchical nature of real-world concepts. For instance, while an LMM might identify a "dog," it struggles to consistently place it within a hierarchy like "animal -> mammal -> canine -> dog." This lack of "hierarchical visual recognition" (HVR) consistency means their understanding is often flat and lacks the nuanced, tree-like structure we humans naturally use. Current LMMs are primarily trained with language modeling objectives, which don't inherently enforce this kind of structured knowledge. The paper aims to bridge this gap by introducing a regularization technique that explicitly guides LMMs to learn and respect taxonomic hierarchies.

Methods

The core of HiR² is a two-pronged regularization approach applied to the intermediate representations of LMMs:

  1. Semantic-Aware Visual Tree Construction: This involves extracting visual features that align with different levels of a taxonomy. It uses a cross-attention mechanism where textual cues from each taxonomic level query visual token embeddings from the LMM's internal layers. This creates "hierarchy-aware" visual features.
  2. Hierarchical Representation Regularization: This is where the novel losses come in:
  3. Taxonomic Entailment Loss: This loss operates in hyperbolic space (specifically, the Lorentz model), which is well-suited for representing hierarchical data due to its exponential volume growth. It enforces that "child" concepts (e.g., "flower") are contained within the "cone" of their "parent" concepts (e.g., "plant"). This ensures the radial hierarchy is respected.
  4. Discriminative Dispersive Loss: To prevent sibling concepts (e.g., different types of flowers) from collapsing into similar representations, this loss operates on a unit hypersphere. It encourages angular separation between sibling embeddings, promoting fine-grained discrimination without disturbing the radial hierarchy established by the entailment loss.

HiR² is designed as a "plug-and-play" module, meaning it can be added to existing LMMs and fine-tuning pipelines with minimal computational overhead and no additional learnable parameters.

Datasets

The paper evaluates HiR² on the following datasets:

  • iNaturalist-2021 (iNat21): This dataset is split into two taxonomies: Plant and Animal, with six hierarchical levels each. It's used for extensive evaluation of HiR²'s effectiveness.
  • CUB-200-2011 (CUB-200): This dataset features a four-level bird taxonomy and is used to demonstrate HiR²'s generalizability across different taxonomic structures.

The experiments are conducted using both supervised fine-tuning (SFT) and dynamic fine-tuning (DFT) paradigms.

Key Insights and Ideas

  • Hyperbolic Geometry for Hierarchy: The paper highlights that hyperbolic geometry is a natural fit for representing hierarchical data because its exponential volume growth mirrors the branching nature of taxonomies, making it ideal for encoding parent-child relationships.
  • Spherical Space for Discrimination: By moving sibling discrimination to a spherical space, the authors cleverly decouple it from the radial hierarchy encoded in hyperbolic space. This prevents interference and allows for improved fine-grained distinctions without disrupting the overall taxonomic structure.
  • Plug-and-Play Effectiveness: The fact that HiR² can be easily integrated into existing LMMs and fine-tuning methods without requiring new parameters is a significant practical advantage, suggesting broad applicability.
  • Complementary Losses: The combination of taxonomic entailment (for hierarchy) and discriminative dispersive loss (for sibling distinction) is crucial. Neither loss alone is sufficient; their synergy leads to more robust and accurate hierarchical representations.

Key Results

  • Consistent Improvement: HiR² consistently improves performance across various LMMs (Qwen2.5-VL, Intern3.5-VL-1B, LLaVA-OV-1.5-4B, Qwen2-VL) and fine-tuning methods (SFT, DFT) on both iNat-Plant and iNat-Animal datasets.
  • Hierarchical Consistent Accuracy (HCA): HiR² significantly boosts HCA, a strict metric requiring correct predictions at all taxonomic levels. For example, on iNat-Animal with Qwen2.5-VL (SFT), HCA increased from 14.06 to 18.07.
  • Leaf-Level Accuracy: The paper also shows improvements in leaf-level accuracy (Accleaf), indicating that the hierarchical regularization doesn't come at the expense of specific category identification.
  • Generalization: HiR² demonstrates strong generalization capabilities, improving performance on both base and novel classes in base-to-novel evaluations, and showing consistent gains on the CUB-200 dataset.
  • Ablation Studies: Experiments confirm that both the taxonomic entailment loss and the discriminative dispersive loss are essential, and regularizing at the last LLM layer yields the best results.

Limitations and Future Work

  • Computational Overhead: While described as plug-and-play, the paper notes that the regularization terms add computational overhead during fine-tuning.
  • Hyperparameter Tuning: The effectiveness of HiR² depends on the chosen loss coefficients ($\lambda_{ent}$, $\lambda_{disp}$), which require tuning. The paper provides some guidance on optimal values.
  • Future Directions: The authors suggest that their framework's success opens avenues for exploring more sophisticated ways to inject structured knowledge into LMMs, potentially leading to more robust and interpretable models. The theoretical analysis of the dispersive loss also lays groundwork for further research into geometric deep learning for multimodal tasks.

Summary updated

Abstract

Taxonomies provide key information about the semantic relationships between concepts and the inherent organization of vision and language. Despite their impressive capabilities, large multimodal models (LMMs) often lack taxonomic knowledge, leading to low hierarchical visual recognition (HVR) consistency. These models typically only rely on language modeling objectives during fine-tuning and lack explicit taxonomy-aware regularization. To address this, we propose Hierarchical Representation Regularization (HiR$^2$), a simple plug-and-play regularizer that improves hierarchical consistency in LMMs. Specifically, we introduce a semantic-aware visual tree construction framework that extracts coarse-to-fine visual features from intermediate LLM layers guided by textual cues. The regularizer combines two complementary objectives: a taxonomic entailment loss that enforces hierarchy via hyperbolic entailment cones in the Lorentz model, and a discriminative dispersive loss that promotes angular separation of semantically similar embeddings on the unit sphere without disturbing the radial hierarchical structure. Extensive experiments demonstrate that HiR$^2$ effectively captures taxonomic structures across diverse LMMs and fine-tuning methods. Code is available at https://github.com/PKUICST-MIPL/HiR2_ICML2026.
BibTeX
@inproceedings{he2026learning,
  author = {Hulingxiao He and Zhi Tan and Yuxin Peng},
  title = {{Learning Taxonomic Trees with Hierarchical Representation Regularization for Large Multimodal Models}},
  booktitle = {International Conference on Machine Learning},
  year = {2026},
  url = {https://openreview.net/forum?id=19Wap8LePy},
}
Figure