Abstract
Robotic grasping in unstructured environments is often hindered by occlusions and the "partial point cloud" problem, where sensors only capture the surfaces facing the camera. This paper introduces YOSO (You Only Scan Once), a novel two-stage pipeline designed to enable high-fidelity 6-DoF grasping of novel objects in dynamic scenes. By performing an initial hemispherical scan to reconstruct 3D meshes and subsequently tracking those objects in real-time, YOSO provides a "completed" scene representation. This approach bridges the gap between static, high-fidelity reconstruction and real-time, partial-view perception, significantly outperforming baseline methods on the GraspNet-1Billion benchmark.
1. Introduction
In real-world robotic manipulation, the environment is rarely static or fully visible. Most contemporary grasping systems rely on depth cameras that provide a single-viewpoint perspective. This results in partial point clouds, leaving the "back" or hidden sides of objects invisible to the robot. If the optimal grasp point is located within these occluded regions, the robot is likely to fail.
Existing solutions typically fall into two categories, both with significant drawbacks. Static Reconstruction methods (like NeRF or TSDF) create detailed 3D maps but cannot adapt if an object is moved. Conversely, Shape Completion methods use deep learning to predict missing geometry; however, these often struggle with accuracy and fail to generalize to novel, complex shapes. YOSO addresses these limitations by creating a system that reconstructs high-fidelity 3D meshes on the fly and maintains their poses dynamically as the workspace changes.
2. The YOSO Pipeline
The YOSO architecture is divided into two distinct operational phases: the Registration Stage and the Dynamic Tracking Stage.
2.1 Stage I: Registration (The "Scan Once")
The process begins with a predefined hemispherical scan of the workspace. During this scan, the system performs two parallel tasks:
- Segmentation: Utilizing XMem, a long-term video object segmentation framework, the system identifies and tracks individual objects across the video stream.
- Reconstruction: Using BundleSDF, the pipeline simultaneously tracks the 6D poses of these objects while generating high-quality 3D meshes. This allows the robot to "learn" the geometry of novel objects without prior CAD models.
2.2 Stage II: Dynamic Tracking and Grasping
Once the initial meshes are generated, the system shifts its computational resources from reconstruction to real-time maintenance and execution:
- Real-Time Tracking: The system continues to track the 6D poses of the reconstructed meshes as they are manipulated or moved within the scene.
- Scene Merging: This is the core innovation of YOSO. The system takes the high-quality meshes from Stage I, transforms them according to their current tracked poses, and merges them with live, partial depth data. This creates a "completed" point cloud that represents the entire object geometry, even for surfaces currently hidden from the camera.
- Grasp Prediction: The completed scene is processed by Scale-balanced GraspNet, which predicts optimal 6-DoF gripper positions based on the full geometry of the objects.
3. Experimental Evaluation
The researchers validated YOSO using the GraspNet-1Billion dataset, a standard benchmark for cluttered scene grasping. To establish a theoretical performance ceiling, the authors extended this dataset to include "Fully Visible" point clouds derived from ground-truth CAD models.
Real-world testing was conducted using a Franka Emika Panda robotic arm equipped with an Intel RealSense L515 LiDAR camera. This setup tested the system's ability to handle physical interactions and sensor noise in real-time.
4. Key Results and Insights
The experimental results highlight the critical importance of scene completeness in robotic grasping:
- Accuracy Improvements: On the GraspNet-1Billion benchmark, YOSO achieved an Average Precision (AP) of 61.22 on "Seen" objects (compared to a 58.95 baseline) and 25.60 on "Novel" objects (up from 22.63).
- Closing the Gap: YOSO significantly narrowed the performance gap between partial-view methods and the theoretical "Fully Visible" upper limit.
- Efficiency: By decoupling mesh generation from pose tracking, the pipeline maintains a functional speed of approximately 2.8 FPS (359.3ms per frame) on an NVIDIA RTX 3090. This is sufficient for interactive tasks where the robot must respond to changes in its environment.
5. Discussion and Limitations
YOSO demonstrates that a reconstructed mesh—even if imperfect—is a superior input for grasp algorithms compared to raw sensor data. Because the system reconstructs meshes during the initial scan, it generalizes effectively to novel objects that were not part of its training set.
However, some limitations remain. The system requires objects to be visible during the initial hemispherical scan; if an object is entirely occluded from the start, it cannot be reconstructed. Additionally, the pipeline is computationally intensive, requiring significant GPU resources to run XMem, BundleSDF, and GraspNet concurrently. Future work will focus on optimizing the speed of the pose tracker and improving robustness in scenarios where objects are moved entirely out of the camera's field of view and later returned.
6. Conclusion
You Only Scan Once (YOSO) provides a robust framework for dynamic 3D scene representation. By combining the strengths of 3D reconstruction with real-time tracking, it allows robots to "see" through occlusions, leading to more reliable and accurate 6-DoF grasping in complex, real-world environments.
Summary updated