Overcoming Fixed Depth Constraints in Hierarchical Representations

19 concepts15 papers1 proposal

The physical and semantic world is inherently compositional. We understand a complex scene by breaking it down into objects, objects into parts, and parts into primitive features. To mirror this, machine learning models frequently employ hierarchical structures—such as trees, scene graphs, and multi-layered networks—to parse images, text, and 3D environments.

However, a fundamental design flaw persists across traditional hierarchical modeling: fixed structural depth. Most architectures assume a static, predefined number of levels (e.g., a rigid three-tier hierarchy of Scene $\rightarrow$ Object $\rightarrow$ Part). This structural inflexibility forces a uniform capacity on data that is naturally variable, creating severe representation bottlenecks.

The Structural Bottleneck of Static Topologies

When a model is constrained to a fixed hierarchical depth, it encounters a dilemma. If a scene is highly complex, containing dozens of interacting entities, a rigid tree architecture quickly runs out of capacity. Conversely, for simple scenes, a deep fixed hierarchy introduces unnecessary redundancy.

As Ting Yao et al. observe in Hierarchy Parsing for Image Captioning, static hierarchical structures fail to capture the fluid complexity of different visual scenes. Forcing a fixed number of levels limits a model's ability to adapt to images that require either highly granular or simplified structural representations.

This limitation becomes acute when models must capture fine-grained details and nuanced relationships. For instance, Fuhai Chen et al. demonstrate in GroupCap: Group-Based Image Captioning With Structured Relevance and Diversity Constraints that when the number of objects in a scene exceeds the capacity of a rigid tree architecture, the model fails to represent all semantic relationships effectively. The same bottleneck appears in cross-modal retrieval; as detailed by Xuri Ge et al. in Structured Multi-modal Feature Embedding and Alignment for Image-Sentence Retrieval, fixed-depth structures struggle to represent complex environments containing numerous interacting entities, limiting the model's capacity to represent relationships beyond a predefined number of semantic nodes.

Fixed Hierarchy (Rigid Bottleneck):
Input Scene ───► [ Level 1: Scene ] ───► [ Level 2: Object ] ───► [ Level 3: Part ]
(If a scene contains deeply nested sub-parts, they are compressed or lost)

Dynamic Hierarchy (Adaptive Depth):
Input Scene ───► [ Level 1 ] ───► [ Level 2 ] ───► [ Level 3 ] ───► [ Level 4 ] ───► [ Level 5 ]
(The hierarchy expands or contracts based on the structural complexity of the input)

Modality-Specific Failures

The consequences of fixed-depth constraints manifest across diverse data modalities:

  • Geometric and 3D Data: In 3D shape analysis, objects possess highly diverse geometries. Elongated objects may require deeper structural decompositions than compact ones. Amir Hertz et al. point out in PointGMM: A Neural GMM Network for Point Clouds that architectures relying on fixed tree depths and branching factors lack the flexibility to adapt to varying geometric complexities. Similarly, Seokhun Choi et al. note in Click-Gaussian: Interactive Segmentation to Any 3D Gaussians that assuming a fixed number of hierarchical levels prevents segmentation frameworks from capturing the full scale of details across varying object boundaries.
  • Temporal Sequences: In video understanding, a narrative can range from a short, simple clip to a long, intricate sequence. Feng Mao et al. argue in Hierarchical Video Frame Sequence Representation with Deep Convolutional Graph Network that a static number of abstraction layers is fundamentally insufficient for long, complex video narratives and overly redundant for short ones.
  • Text and Taxonomies: Language naturally organizes into deeply nested taxonomies. Yet, as Qian Zhang et al. show in Document Hashing with Multi-Grained Prototype-Induced Hierarchical Generative Model, many document modeling approaches restrict themselves to a fixed two-tier (coarse and fine-grained) structure, preventing the system from capturing multi-layered, deeply nested textual taxonomies.

The Path to Dynamic and Continuous Hierarchies

To resolve these limitations, recent research focuses on models that dynamically infer their structural topology directly from the data. Instead of forcing the input into a rigid container, the container reshapes itself to match the input.

Dynamic Structural Reconfiguration

An early foundation for this shift was established by Nicholas J. Adams et al. in Dynamic Trees: Learning to Model Outdoor Scenes, where the authors proposed learning both the model's parameters and its tree structure simultaneously. By allowing the hierarchy to adapt its architecture to the image content, they bypassed the "blocky" segmentation artifacts characteristic of fixed-architecture models. In object modeling, Adam Kortylewski et al. (Greedy Compositional Clustering for Unsupervised Learning of Hierarchical Compositional Models) demonstrated that learning the hierarchical graph structure dynamically allows models to efficiently represent objects with highly diverse geometries.

Unsupervised Depth Discovery

In unsupervised scene decomposition, determining the appropriate level of detail remains a core challenge. Fei Deng et al. address this in Generative Scene Graph Networks, arguing that because the depth of part-whole relationships is fluid and varies significantly between entities, models must adaptively decide the depth of a scene graph based on visual complexity. This insight is shared by Aleksandar Stanic et al. in Hierarchical Relational Inference, where the authors demonstrate the power of dynamically inferring the optimal number of levels required to represent physical interactions in varying environments.

Geometric Re-framing via Hyperbolic Spaces

Perhaps the most mathematically elegant solution to the fixed-depth problem involves changing the underlying geometry of the embedding space. Standard Euclidean spaces struggle to represent hierarchical trees without exponentially increasing dimensions.

To solve this, Hyeongjun Kwon et al. introduced Improving Visual Recognition with Hyperbolical Visual Hierarchy Mapping. By mapping visual features into hyperbolic space—a geometry where space expands exponentially, naturally mirroring the growth of tree branches—the authors bypass the need to predefine a tree's depth or width.

Euclidean Space (Rigid Grid)         Hyperbolic Space (Poincaré Disk)
┌───┬───┬───┐                        The volume of space expands exponentially
├───┼───┼───┤                        toward the boundary, allowing trees of
├───┼───┼───┤                        arbitrary depth and width to embed naturally
└───┴───┴───┘                        without structural crowding.

In hyperbolic space, hierarchical relationships are modeled continuously rather than discretely. A model can place an embedding at varying distances from the origin to represent its depth in a taxonomy, providing the structural flexibility needed to represent both simple objects and highly complex environments within a single, unified framework.

Key Foundational Work

Dynamic Trees: Learning to Model Outdoor Scenes (2002) — contributed 1 concept elements to this direction.
Integrating Grammar and Segmentation for Human Pose Estimation (2013) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_fixed_hierarchy_depth_4e037d21,
  title = {fixed hierarchy depth},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/fixed-hierarchy-depth/rdr_4e037d218cf64c2d9dedd2aabbf746ff},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_fixed_hierarchy_depth_4e037d21_v1,
  title = {fixed hierarchy depth},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/fixed-hierarchy-depth/rdr_4e037d218cf64c2d9dedd2aabbf746ff/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake