Multi-Object Graph Affordance Network: Enabling Goal-Oriented Planning through Compound Object Affordances

Gülsüm Tuba Çibuk Girgin, Emre Ugur

Introduction

In robotics, "affordances" represent the latent action possibilities offered by an environment. While current models excel at single-object or pairwise interactions, they often fail when faced with "compound objects"—complex structures where the addition of a new part dynamically alters the overall affordances. For instance, placing a lid on a cup removes its "insertability." Tuba Girgin and Emre Uğur introduce the Multi-Object Graph Affordance Network (MOGAN) to address these challenges, enabling robots to reason about multi-part structures and plan sequences to achieve complex goals.

Methodology: The MOGAN Framework

MOGAN utilizes Graph Neural Networks (GNNs) to model the physical dependencies within an arbitrary number of objects. The pipeline consists of four primary stages:

  1. Object Encoding: Depth images of individual objects are processed by a pretrained Autoencoder, compressing them into latent vectors that capture geometric features like size and concavity.
  2. Graph Representation: The compound structure is modeled as a graph where nodes represent objects and edges denote spatial relationships (e.g., "stacked on").
  3. Architecture: A GNN processes the graph to generate a state representation of the compound. This is concatenated with the features of a "queried" object to predict the outcome of an interaction.
  4. Effect Prediction: The model predicts three continuous effects: height change ($E_1$), lateral displacement or visibility ($E_2$), and stability ($E_3$).
The MOGAN architecture processes depth images into a graph representation to predict the physical effects of adding new objects to a compound structure.

Experimental Setup

The model was trained on 5,000 data points generated in the PyBullet physics simulator using a diverse inventory of poles, balls, cubes, rings, and cups. Validation was performed on a UR10 robot arm using physical plastic toys to test sim-to-real transfer.

Results and Planning

MOGAN enables goal-oriented planning through tree search, allowing the robot to determine the optimal sequence of placements for tasks like building a tower of a specific height or creating stable bridges.

MOGAN successfully plans and executes complex multi-step tasks in simulation, such as building towers and stacking objects with varying geometries.

Key findings include:

  • Scalability: MOGAN maintained an 80-90% success rate for towers of five objects, significantly outperforming pairwise baselines which dropped to 10%.
  • Sim-to-Real Transfer: The system achieved a 93% success rate in real-world trials, demonstrating robust generalization from simulation to physical hardware.
  • Structural Reasoning: The model successfully reasoned about complex geometries, such as using rings on poles or balancing bridges across two supports.

Conclusion

MOGAN demonstrates that graph-based representations are highly effective for reasoning about the dynamic affordances of compound objects. While current limitations include the assumption of rigid bodies and a lack of integrated gripper geometry, future work involving point clouds and RGB data promises to further enhance the robot's ability to interact with the complex physical world.

Summary updated

BibTeX
@article{girgin2025multi,
  author = {Gülsüm Tuba Çibuk Girgin and Emre Ugur},
  title = {{Multi-Object Graph Affordance Network: Enabling Goal-Oriented Planning through Compound Object Affordances}},
  journal = {IEEE Trans. Cogn. Dev. Syst.},
  year = {2025},
  doi = {10.1109/tcds.2024.3520086},
}
Figure