Introduction
In robotics, "affordances" represent the latent action possibilities offered by an environment. While current models excel at single-object or pairwise interactions, they often fail when faced with "compound objects"—complex structures where the addition of a new part dynamically alters the overall affordances. For instance, placing a lid on a cup removes its "insertability." Tuba Girgin and Emre Uğur introduce the Multi-Object Graph Affordance Network (MOGAN) to address these challenges, enabling robots to reason about multi-part structures and plan sequences to achieve complex goals.
Methodology: The MOGAN Framework
MOGAN utilizes Graph Neural Networks (GNNs) to model the physical dependencies within an arbitrary number of objects. The pipeline consists of four primary stages:
- Object Encoding: Depth images of individual objects are processed by a pretrained Autoencoder, compressing them into latent vectors that capture geometric features like size and concavity.
- Graph Representation: The compound structure is modeled as a graph where nodes represent objects and edges denote spatial relationships (e.g., "stacked on").
- Architecture: A GNN processes the graph to generate a state representation of the compound. This is concatenated with the features of a "queried" object to predict the outcome of an interaction.
- Effect Prediction: The model predicts three continuous effects: height change ($E_1$), lateral displacement or visibility ($E_2$), and stability ($E_3$).
Experimental Setup
The model was trained on 5,000 data points generated in the PyBullet physics simulator using a diverse inventory of poles, balls, cubes, rings, and cups. Validation was performed on a UR10 robot arm using physical plastic toys to test sim-to-real transfer.
Results and Planning
MOGAN enables goal-oriented planning through tree search, allowing the robot to determine the optimal sequence of placements for tasks like building a tower of a specific height or creating stable bridges.
Key findings include:
- Scalability: MOGAN maintained an 80-90% success rate for towers of five objects, significantly outperforming pairwise baselines which dropped to 10%.
- Sim-to-Real Transfer: The system achieved a 93% success rate in real-world trials, demonstrating robust generalization from simulation to physical hardware.
- Structural Reasoning: The model successfully reasoned about complex geometries, such as using rings on poles or balancing bridges across two supports.
Conclusion
MOGAN demonstrates that graph-based representations are highly effective for reasoning about the dynamic affordances of compound objects. While current limitations include the assumption of rigid bodies and a lack of integrated gripper geometry, future work involving point clouds and RGB data promises to further enhance the robot's ability to interact with the complex physical world.
Summary updated