SPOTS: Stable Placement of Objects with Reasoning in Semi-Autonomous Teleoperation Systems

Joonhyung Lee, Sangbeom Park, Jeongeun Park, Kyungjae Lee, Sungjoon Choi

Introduction

In robotic manipulation, the "pick-and-place" cycle is a fundamental operation. While significant progress has been made in autonomous grasping ("picking"), the "placing" phase remains a bottleneck. Effective placement requires more than just avoiding collisions; it necessitates a combination of physical stability and semantic appropriateness. For instance, a robot should not only ensure a book won't fall off a shelf but also that it is placed among other books rather than on a rack of fragile glassware.

The SPOTS pipeline: from real-world sensing to physics-based stability verification and LLM-driven receptacle reasoning.

The SPOTS (Stable Placement of Objects with Reasoning in Semi-Autonomous Teleoperation Systems) framework addresses this by integrating physics-based simulations with Large Language Models (LLMs). This hybrid approach allows a semi-autonomous system to suggest placement candidates that are both physically robust and contextually "sane" to a human operator.

Methodology

The SPOTS architecture follows a two-stream pipeline to evaluate potential placement locations.

Physics-Based Stability Verification

The system first creates a "digital twin" of the environment. Using RGBD data, it performs open-vocabulary object detection via OWL-ViT and pose estimation with AprilTags. This scene is reconstructed in the MuJoCo physics simulator. Within this simulation, the system performs "test-placements" across a grid of coordinates. By injecting small physical perturbations (e.g., vibrations), SPOTS identifies a set of physically stable candidate points ($P_s$) where the object is unlikely to tip or fall.

Receptacle Reasoning

Simultaneously, the system leverages the semantic knowledge of an LLM (GPT-3.5-turbo). The LLM is provided with object labels from the scene and user-defined preferences (e.g., "organize by category"). It then identifies the most logical "receptacle"—such as a specific shelf or container—for the target object. This step ensures the robot's actions align with human common sense and organizational intent.

Decoupling stability verification (left) from semantic receptacle reasoning (right).

Reward-Guided Sampling

The outputs from both streams are fused using Kernel Density Estimation (KDE) to generate a unified probability distribution or "heat map." A tunable parameter, $\beta$, allows operators to balance the importance of physical stability versus semantic reasoning. The system then presents the highest-probability candidates to the user for final selection.

Evaluation and Results

The framework was evaluated across three simulated environments—Dish Rack, Bookshelf, and Category sorting—using assets from the YCB and Google Scanned Objects datasets. Real-world validation was conducted using a 6-DoF robot arm and common household items.

Key findings include:

  • Performance: SPOTS achieved stability and reasoning success rates between 0.85 and 0.95, consistently outperforming baselines like Code-as-Policies (CaP) and Language-to-Reward.
  • Efficiency: By using LLMs for high-level region identification rather than low-level code generation, SPOTS is significantly more token-efficient. In the bookshelf task, it required only 323 tokens compared to 985 for CaP, resulting in faster inference.
  • Robustness: The physics-based stream proved vital in constrained environments like dish racks, where purely language-based or geometric heuristics often fail to account for precarious balance.

Discussion and Future Work

A core insight of SPOTS is the effective decoupling of physics and logic. While LLMs excel at "common sense" reasoning, they often struggle with precise 3D spatial calculations. By delegating the "math" to a physics engine and the "logic" to an LLM, SPOTS leverages the strengths of both paradigms.

Current limitations include a dependency on pre-existing CAD models for the "real-to-sim" transition and the lack of a closed-loop feedback mechanism during the physical execution of the placement. Future research may explore using Neural Radiance Fields (NeRF) for on-the-fly reconstruction of unknown objects and integrating real-time tactile or visual feedback to handle slippage during placement.

Summary updated

BibTeX
@inproceedings{lee2024spots,
  author = {Joonhyung Lee and Sangbeom Park and Jeongeun Park and Kyungjae Lee and Sungjoon Choi},
  title = {{SPOTS: Stable Placement of Objects with Reasoning in Semi-Autonomous Teleoperation Systems}},
  booktitle = {IEEE International Conference on Robotics and Automation},
  year = {2024},
  doi = {10.1109/icra57147.2024.10611613},
}
Figure