Machine learning relies heavily on hierarchical representations. Whether organizing text into words, clauses, and sentences, or decomposing a 3D shape into surfaces and fine edges, hierarchies allow models to mirror the multi-scale structure of the real world.
However, a fundamental limitation persists across traditional hierarchical models: structural rigidity. In most architectures, the depth of the hierarchy—the number of layers, scales, or abstraction tiers—is a fixed hyperparameter configured manually before training. This "one-size-fits-all" approach assumes that all data points or tasks share the same underlying complexity, forcing shallow representations on intricate structures or wasting computation on redundant layers for simple ones.
To overcome this, a growing body of research focuses on automated hierarchy discovery and adaptive depth. By transitioning from static configurations to data-driven, self-organizing structures, models can dynamically scale their capacity and resolution to match the inherent complexity of the input.
The Cost of Static Hierarchies
When the depth of a hierarchy is fixed, it acts as a rigid constraint on how a model processes information. Across diverse domains, researchers have documented the distinct bottlenecks caused by this limitation:
- Spatial and Geometric Embeddings: In spatial partitioning systems like the Nested Barycentric Coordinate System (NBCS) proposed by Lee-Ad Gottlieb et al., determining the right level of resolution traditionally requires extensive, computationally expensive cross-validation pipelines. Similarly, in 3D shape representation, Hao Huang et al. observe that fixing the number of decomposition scales fails to align with objects that mix highly complex geometries with completely flat surfaces.
- Reinforcement Learning and Planning: Hierarchical reinforcement learning agents often operate within rigid, pre-defined policy tiers. As Andrew Levy et al. note, a fixed number of layers can be highly redundant for straightforward tasks yet entirely insufficient for long-horizon, complex environments. Pierre-Luc Bacon et al. argue that requiring human experts to pre-specify the number of available sub-policies limits an agent's ability to autonomously adapt to newly discovered environmental structures.
- Sequential and Textual Data: Language and music naturally vary in structural density. Wanshan Zheng et al. point out that static text classification hierarchies cannot adapt to the varying lengths and clause structures of different documents. The authors of Hierarchical Recurrent Neural Networks for Conditional Melody Generation with Long-term Structure similarly emphasize that fixed tiers struggle to capture musical compositions that shift between simple repetitions and intricate arrangements.
Nonparametric and Information-Theoretic Discovery
One prominent strategy for achieving adaptive depth is to let mathematical priors or information-theoretic boundaries dictate the structure of the hierarchy automatically. Instead of human tuning, the model uses the statistical properties of the dataset to find its natural scale.
Bayesian Nonparametrics
Bayesian nonparametric frameworks allow models to grow both horizontally (width) and vertically (depth) as they encounter more data. For example, XianXing Zhang et al. combined a nested Chinese restaurant process with a tree-structured shrinkage prior to build a factor model that infers the optimal number of clusters and factors per cluster simultaneously. In time-series analysis, Dan dan Guo et al. demonstrated that stacking latent layers via nonparametric approaches allows deep dynamical systems to automatically determine the optimal depth needed to capture long-range dependencies.
Compression and Entropy Bottlenecks
Another approach treats hierarchy discovery as a data compression problem. Mete Özay leveraged the Minimum Description Length (MDL) principle to automatically determine the depth of a compositional shape hierarchy, ensuring the resulting representation is neither over-simplified nor unnecessarily deep.
For neural networks, compression can be enforced via information bottlenecks. Melika Behjati et al. applied a Nonparametric Variational Information Bottleneck (NVIB) across stacked layers, allowing a character-level Transformer to dynamically determine the necessary depth of abstraction to group characters into linguistic units without predefined tokenizers. In graph learning, Junran Wu et al. utilized structural entropy to minimize structural uncertainty, allowing a model to automatically detect the natural hierarchical depth of a graph rather than treating it as a manual hyperparameter.
Dynamic Routing and Search
Rather than relying purely on statistical priors to establish a global hierarchy, some systems evaluate data complexity on the fly, dynamically adding, removing, or routing through hierarchical layers during training or inference.
[Input Data]
│
▼
┌────────────────────────────────────────┐
│ Dynamic Routing Mechanism │
│ (Evaluates Local/Task Complexity) │
└────────────────────────────────────────┘
│
├─► Simple Input ──► [Truncate Hierarchy / Coarse Grid]
│
└─► Complex Input ──► [Expand Hierarchy / Deep Tree]
In continual learning, where models must learn new tasks sequentially without forgetting old ones, organizing adapters hierarchically helps balance general knowledge sharing with task specialization. Yu-Yang Qian et al. introduced a bandit-based search algorithm to guide layer-wise adapters, automating tree-depth selection to optimize training speed and accuracy based on task similarity.
This dynamic flexibility is equally vital for reinforcement learning agents operating in unpredictable environments. Paul Mattes et al. developed a world model framework that allows for the automatic addition or removal of hierarchical layers. This mechanism enables the agent to dynamically optimize its planning horizons, expanding its hierarchy to imagine complex, long-term trajectories or truncating it when navigating simpler environments.
Conclusion
The shift toward automated hierarchy discovery represents a fundamental evolution in architecture design. By moving away from rigid, manually configured structural hyperparameters and embracing adaptive depth—whether through nonparametric Bayesian priors, information-theoretic bottlenecks, or dynamic routing algorithms—models gain the flexibility to self-organize. Ultimately, this allows systems to match the true, fluid granularity of the real world, making representations both more efficient and structurally precise.
Key Foundational Work
Hierarchical Part-Based Visual Object Categorization (2005) — contributed 1 concept elements to this direction.
Unsupervised Image Classification with a Hierarchical EM Algorithm (1999) — contributed 1 concept elements to this direction.