VISO-Grasp: Vision-Language Informed Spatial Object-centric 6-DoF Active View Planning and Grasping in Clutter and Invisibility

Yitian Shi, Di Wen, Guanqi Chen, Edgar Welte, Sheng Liu, Kunyu Peng, Rainer Stiefelhagen, Rania Rayyes

Abstract

Robotic grasping in unstructured environments remains a significant challenge due to severe occlusion and the potential for target objects to be entirely hidden from initial view. This paper presents VISO-Grasp, a novel framework that integrates Vision-Language Models (VLMs) with active camera planning and real-time grasp fusion. By leveraging the high-level spatial reasoning of VLMs, the system identifies relational hierarchies between objects to guide a velocity-field-based Next-Best-View (NBV) planner. Furthermore, a Bayesian grasp fusion mechanism utilizing von Mises-Fisher (vMF) distributions ensures robust 6-DoF grasp execution by aggregating multi-view predictions. Experimental results in cluttered real-world scenarios demonstrate a high success rate (87.5%) and superior performance in retrieving invisible targets compared to static and traditional NBV baselines.

Introduction

In environments such as cluttered warehouses or domestic kitchens, target objects are frequently obscured by other items. For a robot to successfully interact with these objects, it must possess the ability to reason about spatial relationships and actively adjust its viewpoint to "uncover" the target. Traditional robotic systems often rely on static viewpoints or pre-defined paths, which are insufficient when a target is buried or heavily occluded.

VISO-Grasp addresses these limitations by treating grasping as an active, perception-driven task. It mimics human-like behavior by using high-level reasoning to understand which obstacles must be bypassed or moved to reveal a goal. By combining state-of-the-art foundation models with a continuous motion planner and a probabilistic grasp fusion module, VISO-Grasp enables reliable 6-DoF grasping in the "wild."

Methodology

The VISO-Grasp framework is composed of three interconnected modules designed to handle perception, planning, and execution.

1. AMOV3D: Adaptive Multi-view Open-Vocabulary 3D Object Detector

The perception layer utilizes a VLM-based pipeline (Qwen2.5-VL) integrated with Grounding DINO and Segment Anything 2 (SAM2). Unlike standard detectors that merely label objects, AMOV3D generates a "relational hierarchy." It categorizes surrounding items as "High" (occluding), "Low" (non-obstructing), or in "Proximity" to the target. This spatial context allows the robot to understand the physical structure of the clutter and identify which areas of the scene are likely hiding the target.

2. TGV-Planner: Target-Guided View Planner

When a target is occluded, the system employs a Target-Guided View (TGV) Planner. This module utilizes a Velocity-field-based Next-Best-View (V-NBV) approach. Rather than sampling discrete, disconnected viewpoints, the planner treats the camera's trajectory as a particle moving through a flow field. This field is shaped by the spatial relationships identified by the VLM, guiding the camera toward angles that maximize target visibility while maintaining smooth, continuous robot motion.

3. RT-UMGF: Real-Time Uncertainty-guided Multi-view Grasp Fusion

To ensure grasp stability, VISO-Grasp does not rely on a single sensor snapshot. The RT-UMGF module fuses 6-DoF grasp candidates from multiple viewpoints at a frequency of 10Hz. It employs von Mises-Fisher (vMF) distributions to perform Bayesian updates on the grasp poses. This probabilistic approach allows the system to "average" orientation and position predictions while explicitly accounting for directional uncertainty, effectively filtering out sensor noise and model hallucinations.

Experimental Evaluation

The system was evaluated across eight distinct cluttered scenes involving common household objects. The experiments were designed to test varying degrees of difficulty, ranging from partial occlusion to total invisibility.

Key Results

  • High Success Rate: VISO-Grasp achieved an average Final Success (FS) rate of 87.5% across all test scenarios.
  • Efficiency in Motion: The TGV-Planner outperformed traditional NBV baselines (such as Breyer’s planner) by reaching the target with the fewest required grasp attempts and more efficient trajectories.
  • Handling Invisibility: In a specific "Green Pear" test case where the target was completely hidden, VISO-Grasp maintained a 100% success rate. In contrast, static viewpoint baselines failed nearly 50% of the time as they could not adapt to the lack of initial visual data.
  • Ablation Insights: Removing the Real-Time Multi-view Grasp Fusion (RT-UMGF) significantly degraded performance, confirming that integrating data across multiple views is essential for robust grasping in dense clutter.

Discussion and Limitations

While VISO-Grasp demonstrates significant advancements in active grasping, two primary limitations remain. First, the reliance on large-scale VLMs introduces a computational bottleneck; the latency of high-resolution image processing currently limits the system's real-time responsiveness. Second, the framework assumes a quasi-static environment. If the scene changes dynamically (e.g., objects shifting or falling) independently of the robot's actions, the spatial map may become outdated.

Conclusion

VISO-Grasp represents a step forward in autonomous robotic manipulation by bridging the gap between high-level semantic reasoning and low-level geometric planning. By treating the robot as an active observer capable of reasoning about its environment, the system achieves high reliability in scenarios that typically baffle static perception systems. Future work will focus on optimizing VLM inference speeds and extending the framework to handle highly dynamic, non-static environments.

Summary updated

BibTeX
@inproceedings{shi2025viso,
  author = {Yitian Shi and Di Wen and Guanqi Chen and Edgar Welte and Sheng Liu and Kunyu Peng and Rainer Stiefelhagen and Rania Rayyes},
  title = {{VISO-Grasp: Vision-Language Informed Spatial Object-centric 6-DoF Active View Planning and Grasping in Clutter and Invisibility}},
  booktitle = {IEEE/RSJ International Conference on Intelligent Robots and Systems},
  year = {2025},
  doi = {10.1109/iros60139.2025.11246329},
}
Figure