Predicting Stacking Hierarchies for Robotic Manipulation in Clutter

33 concepts20 papers1 proposal

When humans look at a messy desk or a crowded sink, we don't just see a collection of isolated items. We instantly perceive a physical hierarchy: we know that the coffee mug is resting on top of a notebook, and a pen is wedged precariously underneath both. If we want to retrieve the notebook, we intuitively lift the mug first to prevent a spill.

For robots, however, this task has traditionally been treated as a flat geometric problem. Conventional robotic grasping and bin-picking systems often assume that objects are arranged in stable, non-overlapping configurations on a flat surface (Yang et al.). When these systems encounter real-world clutter—such as casually stacked boxes on a shipping pallet or tangled piles of industrial parts—they treat all visible surfaces as equally accessible.

The result of this blind spot is physical failure. Attempting to extract a wedged or buried object directly can cause structural collapse, mechanical damage, or environmental spillage (Tchuiev et al.; Gilles et al.). To operate safely in unstructured environments, robots must shift from purely localized object detection to stacking order reasoning—modeling the physical and spatial dependencies between objects as a structured hierarchy.

The Core Intuition: Scenes as Directed Graphs

The fundamental breakthrough in this research direction is the formalization of a cluttered scene as a directed graph or tree rather than a set of independent 3D coordinates.

Imagine a pile of three overlapping books: A is on top of B, which is on top of C. Instead of merely identifying the bounding boxes of A, B, and C, relational models predict "parent-child" or "support-dependency" relationships.

   [ Book A ]  (Top-most / Accessible)
       │
       ▼
   [ Book B ]  (Obstructed by A)
       │
       ▼
   [ Book C ]  (Buried / Base Support)

By organizing the scene into this structural representation, the robot can perform a topological sort to find a valid sequence of actions. Nodes with no incoming dependencies (or parents) represent objects that are completely unblocked and can be safely moved without disturbing the rest of the stack (Zhang et al.).

This graph-centric framing transforms a chaotic physical problem into a logical sequence of removals, allowing a robot to systematically clear obstacles to reach a deeply buried target (Zhang et al.; Xu et al.).

How Robots Learn to See "Who is on Top"

Mapping these physical boundaries is challenging because high levels of clutter and object overlap obscure the structural lines between individual items (Rabino et al.). Researchers have developed several distinct mechanisms to infer these hidden hierarchies from visual data.

1. Relational Neural Networks and Matrices

One approach is to extend standard object detection architectures to output relational data alongside bounding boxes. For example, Park et al. modified a single multi-task network to predict parent and child relationships within grid cells, constructing a directed relationship graph. Interestingly, the authors observed that computer vision models identify the top-most object in a stack with significantly higher accuracy than the supporting object underneath. Prioritizing the detection of these top-level relationships provides a more reliable foundation for planning manipulation sequences.

Other frameworks encode these hierarchies globally. Gilles et al. utilize a relationship matrix that explicitly categorizes objects into distinct layers (such as top or secondary). To handle the inherent ambiguity of single-perspective views, Tchuiev et al. introduce a probabilistic adjacency matrix that fuses information across multiple camera views, allowing the robot to quantify its uncertainty before attempting a grasp.

2. Attention Mechanisms and Transformers

Because stacking relationships are inherently relational and non-local, Transformers and Graph Neural Networks (GNNs) are natural fits for this domain. Rabino et al. introduced an end-to-end transformer-based model called D3G designed specifically to map parent-child boundaries in dense bins. Similarly, Kulshrestha et al. leverage graph attention networks to capture multi-level support hierarchies during structural rearrangement tasks. For reinforcement learning agents, Haramati demonstrates that an entity-centric Transformer architecture allows a robot to implicitly model relational constraints from raw pixels, generalizing from a few objects to complex scenes containing more than ten.

3. Grounding via Language and Vision-Language Models

A recent frontier leverages the zero-shot spatial reasoning of large Vision-Language Models (VLMs). In the FreeGrasp pipeline, Fasoli et al. use a "Set-of-Mark" visual prompting strategy to label objects with visual markers, allowing a VLM to textually resolve spatial dependencies and explicitly name which obstructing items must be removed first. In a similar vein, Song et al. process natural language scene descriptions through BERT to identify "surface attributes," ensuring the robot targets only the exposed top layers of a stack to maintain collision-free trajectories.

[Raw Image] ──> [Set-of-Mark Prompting] ──> [VLM Reasoning] ──> [Precedence List]

From Hierarchy to Action: Why It Works

Predicting the graph is only half the battle; the robot must translate this hierarchy into physical actions. Stacking order reasoning drastically improves manipulation success because it bridges the gap between abstract geometric planning and physical reality in three ways:

  • Preventing Structural Collapse: In structured environments like industrial pallets, extracting an item from the bottom can cause the entire stack to topple. By combining 6DoF pose estimation with mask visibility and height data, pipelines can isolate the most graspable, unburdened objects at the top of a stack to preserve equilibrium (Jeevanandam et al.).
  • Optimizing the Sequence Length: Representing dependencies as a tree allows planners to minimize redundant moves. Wu et al. propose a Stacking Relationship Tree (SRT) that distinguishes between "stable" and "weak" support relationships. This distinction allows a planner to recognize when it can safely lift an entire sub-tree (multiple stacked objects) simultaneously, reducing execution time by nearly 30%.
  • Handling Non-Rigid and Dynamic Clutter: For highly complex objects like intertwined cables, calculating exact physics is impossible. Zhang (Lei) et al. show that simply weighting grasp selection toward the highest vertical points of the clutter allows a robot to systematically peel away top layers, avoiding entanglement. When pushing is required to clear space, Boularias et al. demonstrate that decomposing scene dynamics into direct robot-object interactions and secondary object-object collisions allows a network to accurately predict how a single push will propagate through a dense cluster.

Ultimately, explicitly modeling physical obstructions and stacking relationships—whether through specialized neural networks combined with formal planners like POMDPs (Zhang (Hanbo) et al.) or through interactive haptic exploration (Sinapov et al.)—transforms robotic manipulation from a series of risky, blind guesses into a deliberate, strategic sequence of actions.

Key Foundational Work

RPRG: Toward Real-time Robotic Perception, Reasoning and Grasping with One Multi-task Convolutional Neural Network (2018) — contributed 4 concept elements to this direction.
Learning to Order Objects Using Haptic and Proprioceptive Exploratory Behaviors (2016) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_stacking_order_reasoning_b6ddad3f,
  title = {stacking order reasoning},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/stacking-order-reasoning/rdr_b6ddad3fa6444f0ea08403beb31c2a11},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_stacking_order_reasoning_b6ddad3f_v1,
  title = {stacking order reasoning},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/stacking-order-reasoning/rdr_b6ddad3fa6444f0ea08403beb31c2a11/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake