Prepare the Chair for the Bear! Robot Imagination of Sitting Affordance to Reorient Previously Unseen Chairs

Xin Meng, Hongtao Wu, Sipu Ruan, Gregory S. Chirikjian

TL;DR

This paper introduces a novel robot manipulation system that allows robots to "imagine" how to prepare previously unseen chairs for sitting. By reconstructing 3D models and simulating interactions, the robot can identify chairs, determine their functional "sitting" pose, and physically reorient them, achieving a high success rate in real-world experiments.

What Problem Does This Paper Address?

As robots increasingly enter human environments, they need to interact intelligently with unfamiliar objects to perform household tasks. A key challenge is understanding an object's affordance – how it can be used. For instance, a chair's "sitting" affordance is only available when it's in a stable, upright, and accessible pose. This paper tackles the problem of enabling a robot to:

  1. Identify if an unknown object is a chair.
  2. Determine the correct pose (functional pose) for it to afford sitting.
  3. Physically manipulate the object into that pose, even if it's initially in a random, occluded, or unusable orientation. This is particularly challenging because the robot has no prior knowledge of the object's category or its initial pose.

Methods

The core of the proposed method is "robot imagination," which leverages physics simulations to reason about object affordances. The pipeline involves several key steps:

  • Object Reconstruction: When an object is placed on a surface, parts of it are occluded. To get a complete 3D model, the robot first finds "stable poses" that expose the occluded parts. It then manipulates the object to one of these poses and scans it again, fusing the data to create a complete model. This process is guided by understanding object stability and "perceptibility" (how much of the object is visible).
  • Sitting Imagination: Using the complete 3D model, the robot simulates a virtual agent (a teddy bear, representing humans) interacting with the object in various stable poses. It evaluates the success of the simulated sitting based on criteria like joint angles, contact points, symmetry, and height. This simulation helps classify the object as a chair and identify its functional pose.
  • Functional and Perceptible Equivalence: The paper introduces the concept that many object poses are functionally and perceptibly equivalent. This means that if an object can afford sitting in one pose, it can in others within the same "coset" (a mathematical grouping of poses). This simplifies the search for functional poses.
  • Robot Manipulation: Once the functional and accessible pose is identified through imagination, a robot arm physically reorients the chair. The system also plans motion for a humanoid robot to approach and seat the teddy bear, ensuring the prepared chair is accessible.

Datasets

  • Calibration Set: The imagination model was calibrated using 30 synthetic chairs and one real chair.
  • Test Set: The experiments were conducted on 7 real chairs and 6 non-chair objects, all designed for young children. These objects were previously unseen by the robot.

Key Insights and Ideas

  • "Robot Imagination" as a Core Capability: The paper strongly advocates for using physics simulation as a form of "imagination" for robots to understand object affordances, moving beyond simple classification to functional reasoning.
  • Stability and Perceptibility Guide Reconstruction: The idea that finding stable poses can reveal occluded parts is crucial for accurate 3D reconstruction of objects in real-world scenarios.
  • Functional Equivalence Simplifies Search: Recognizing that many poses are functionally the same significantly reduces the search space for identifying the correct object orientation.
  • Teddy Bear as a Proxy: Using a teddy bear as a stand-in for human users effectively demonstrates the robot's ability to prepare objects for interaction, bridging the gap between simulation and real-world application.

Key Results

  • Chair Classification: The robot achieved a 100% success rate in classifying objects as chairs or non-chairs across 45 trials.
  • Chair Preparation: The robot successfully prepared chairs into functional and accessible poses in all 30 trials involving chairs.
  • Teddy Bear Seating: The robot successfully seated the teddy bear on the prepared chairs in 29 out of 30 trials, achieving a 96.7% success rate.
  • Functional Pose Prediction: The proposed method outperformed five baseline methods on functional pose prediction, achieving a 100% success rate compared to the best baseline at 80.0%.

Limitations and Future Work

  • Limitations:
    • The success rate for seating the bear dropped slightly in one case due to a lack of armrests on a specific chair, making the imagined sitting pose difficult for the NAO robot to execute precisely. Control errors and the NAO's workspace limitations also contributed.
  • The current method relies on attaching handles to objects to simplify grasping, which is a simplification for the manipulation task.
  • Future Work:
    • Extend the imagination method to other objects that involve complex human whole-body interactions.
  • Explore more functionality-related physical properties through active interaction.
  • Potentially apply the method to tasks like tidying up chairs or preparing chairs for meetings, and to assist elderly individuals.

Summary updated

Abstract

In this letter, a paradigm for the classification and manipulation of novel objects is established and demonstrated with the example of chairs. Our approach leverages the robot's understanding of object stability, perceptibility, and affordance to prepare previously unseen and randomly oriented chairs on which a teddy bear is to be seated. The teddy bear is a proxy for an elderly person, hospital patient, or child. By autonomously reconstructing a complete model of the object and inserting it into a physical simulator (i.e., the robot's “imagination”), the robot assesses whether or not the object is a chair and, if it is, determines how to reorient it properly to be used. Experimental results show that our method achieves a high success rate on the real robot task of chair preparation. Also, it outperforms several baseline methods on the task of upright pose prediction for chairs. The same methodology can be easily transferred to a wide variety of application scenarios, and illustrates a broader paradigm in affordance-based reasoning.
BibTeX
@article{meng2023prepare,
  author = {Xin Meng and Hongtao Wu and Sipu Ruan and Gregory S. Chirikjian},
  title = {{Prepare the Chair for the Bear! Robot Imagination of Sitting Affordance to Reorient Previously Unseen Chairs}},
  journal = {IEEE Robotics Autom. Lett.},
  year = {2023},
  url = {https://openreview.net/forum?id=1Bvuerpxtj},
}
Figure