Most modern computer vision models treat an image as a flat grid of pixels or a sequence of tokens. While this approach has led to impressive results in classification and detection, it fundamentally differs from how humans perceive the world. We do not see a "bus" as a monolithic blob of pixels; we see a vehicle composed of wheels, windows, and doors, which are themselves composed of smaller geometric primitives.
This research direction focuses on moving beyond flat representations toward part-whole hierarchies, where visual scenes are understood as nested structures of reusable components.
The Limits of Flat Representations
Standard Convolutional Neural Networks (CNNs) and Vision Transformers (ViTs) often suffer from "spatial blindness." As noted by Minh Tran et al., scalar-based neurons struggle to encode the specific instantiation parameters—like pose, scale, and orientation—necessary to maintain structural dependencies. When a model only looks for the presence of features rather than their global arrangement, it can be easily fooled or fail to generalize to new viewpoints.
Arjun Punjabi et al. argue that this lack of holistic representation leads to classifications based on noisy local triggers. Without a structural prior, models struggle with "one-shot learning," a feat humans perform easily by recognizing how a new concept is composed of familiar building blocks. Brenden M. Lake et al. demonstrated that modeling images as the result of a compositional, causal process allows for human-level performance in learning new characters from a single example.
Compositional Grammars and Graphs
One of the earliest successful approaches to this problem involved And-Or Graphs (AOG) and probabilistic grammars. Long Zhu et al. introduced methods to automatically discover object structures by combining the hierarchical flexibility of grammars with spatial constraints.
In this framework:
- OR nodes represent alternative ways a part can appear (e.g., a "head" can be seen from the front or side).
- AND nodes represent the decomposition of a whole into its constituent parts (e.g., a "face" is a "nose" AND "eyes" AND a "mouth").
Tianfu Wu and Xi Song integrated these classical hierarchical models with modern CNNs to "unfold" latent structures, making the model's internal reasoning human-readable. By explicitly modeling how objects are built from parts, these systems provide a structural prior that flat architectures lack.
Dynamic Parsing and GLOM
A major challenge in hierarchical modeling is how to represent a tree-like structure within a fixed-weight neural network. Geoffrey E. Hinton proposed a conceptual architecture called GLOM to solve this. Instead of using explicit pointers or dynamic neurons, GLOM uses "islands of identical vectors."
In GLOM, multiple layers of a network represent different levels of a hierarchy (e.g., pixels, parts, objects). Through a consensus-building process, nearby locations in the image that belong to the same object eventually adopt the same high-level vector. This allows the network to dynamically parse an image into a hierarchy without needing to change its physical connections.
Similarly, Mingyu Ding and Yikang Shen developed DependencyViT, which flips the standard attention mechanism to "send" information. This causes dependency trees to emerge naturally, allowing the model to discover "wheel-to-car" relationships without manual labels.
The Geometry of Hierarchies
If the world is hierarchical, the mathematical space we use to represent it should reflect that. Standard models use Euclidean space, which is "flat." However, Karan Desai and Maximilian Nickel introduced MERU, which embeds images and text into hyperbolic space.
Hyperbolic geometry is naturally suited for trees because its volume grows exponentially with its radius, mirroring how the number of nodes in a hierarchy grows as you move from the root to the leaves. Ziwei Wang et al. showed that learning visual hierarchies in hyperbolic space significantly improves image retrieval by capturing the "containment" relationships between scenes, objects, and parts.
Unsupervised Discovery and Reasoning
Recent work has focused on discovering these hierarchies without human supervision. Shengcao Cao developed HASSOD and SOHES, frameworks that learn to segment objects and their constituent parts using only raw, unlabeled images. By evolving through self-exploration and self-correction, these models learn to perceive scenes with the multi-granularity depth that humans use naturally.
This structural understanding is critical for reasoning. Yining Hong et al. introduced the PTR benchmark, which reveals that while current AI is good at identifying whole objects, it struggles to reason about the geometry and physics of object parts. Bridging this gap is essential for tasks like autonomous driving, where Kai Chen et al. found that hierarchical scene descriptions significantly reduce the omission of critical objects in complex environments.
By moving from flat patterns to structured hierarchies, researchers are building models that don't just recognize pixels, but understand the compositional logic of the physical world.