TL;DR
This paper introduces Direct Contact-Tolerant Motion Planning (DCT), a novel system that uses Vision-Language Models (VLMs) to enable robots to safely and efficiently navigate cluttered environments by intelligently interacting with movable obstacles. DCT achieves this by directly processing point cloud data, overcoming the limitations of traditional methods that rely on indirect representations and struggle with real-time adaptiveness.
What Problem Does This Paper Address?
Autonomous mobile robots often face cluttered environments where traditional collision avoidance, treating all objects as rigid and immovable, leads to inefficient or impossible navigation. Many real-world obstacles, like curtains or empty boxes, are actually movable or deformable and can be safely contacted to clear paths. This challenge, known as Contact-Tolerant Motion Planning (CTMP), requires robots to not only identify these movable objects but also reason about how to interact with them without causing damage or getting stuck. Existing CTMP methods often rely on pre-built maps or simplified obstacle representations, which are inaccurate and lack adaptiveness to dynamic environments. Furthermore, reasoning about an object's movability is complex, depending on the robot's capabilities, the object's properties, and the specific task. This paper aims to bridge this gap by proposing a system that directly uses visual and language understanding to achieve robust and efficient contact-tolerant navigation.
Methods
The proposed system, Direct Contact-Tolerant Motion Planning (DCT), comprises two key modules:
- VLM Point Cloud Partitioner (VPP): This module is responsible for identifying movable obstacles. It leverages VLMs for contact-tolerance reasoning. Raw RGB images are processed by an open-set detector guided by language prompts to identify candidate obstacle regions. A VLM then filters these candidates, determining which are movable based on task-specific prompts. To overcome the latency of VLM inference, VPP employs a memory-driven approach: it caches VLM inference masks along with robot poses. When new sensor data arrives, VPP retrieves past masks, propagates them across frames using odometry and homography, and projects them onto the current lidar scan. This allows for real-time partitioning of the point cloud into "contact-tolerant" (movable) and "contact-intolerant" (fixed) sets. A 3D clustering step further refines these partitions, ensuring 3D continuity and correcting for isolated points or missed returns.
- VPP Guided Navigation (VGN): This module takes the contact-partitioned point cloud from VPP and formulates the motion planning problem. Instead of using indirect distance constraints, VGN directly uses the partitioned point cloud to define constraints. To handle the computational complexity of direct point distance constraints, VGN employs a specialized deep neural network (DNN). This DNN is trained to imitate the solution of a Model Predictive Control (MPC) problem, converting time-consuming iterative computations into fast, feed-forward inference. VGN also includes a "correcting mode" that triggers if a push fails or the robot gets stuck. In this mode, contact points are relabeled as non-movable, the robot reverses to a safe state, and replanning occurs.
Datasets
The paper does not explicitly mention specific datasets used for training the VLMs or the DNN. However, it states that experiments were conducted using:
- Isaac Sim: A high-fidelity simulation platform for robot testing and training, using a simulated Nova Carter robot with lidar and RGB camera.
- Real-world robot: A physical mobile robot equipped with a Livox LiDAR, RGB-D camera, and an Intel NUC.
For evaluation, the authors created three specific obstacle scenarios in Isaac Sim:
- Movable obstacle with a wide side path.
- Movable obstacle with a narrow side path.
- Fixed obstacle with a wide side path.
They also designed mixed cluttered environments with varying combinations of fixed and movable obstacles.
Key Insights and Ideas
- Direct Point Perception for CTMP: The core innovation is moving away from indirect spatial representations (like occupancy grids) towards direct processing of point clouds. This allows for more accurate and adaptive handling of complex obstacle shapes and movability.
- VLM for Movability Reasoning: Leveraging VLMs for contact-tolerance reasoning is a powerful way to imbue robots with nuanced understanding of object properties and task context, going beyond simple geometric classifications.
- Memory-Driven VLM for Real-Time Performance: The temporal memory and mask propagation strategy for VPP is crucial for achieving real-time performance, as it avoids the computational bottleneck of running full VLM inferences at every sensor update.
- Learned Planner for Direct Constraints: Training a DNN to solve the direct point distance constraints bypasses the computational cost of traditional optimization solvers, enabling fast, end-to-end planning.
- Adaptive Correction Mechanism: The "correcting mode" in VGN is vital for robustness, allowing the robot to recover from failed interactions and replan safely.
Key Results
- VLM Performance: GPT-5 achieved perfect precision in identifying pushable obstacles, while Gemini 2.5 had the highest recall. GPT-5 was chosen for its balanced performance and reliability.
- Simulation Performance: In various scenarios, DCT consistently outperformed baselines like NeuPAN and Ellis22 in terms of navigation time and average speed. For instance, in Case 1 (movable obstacle, wide path), DCT achieved the lowest navigation time (4.22s) and highest average speed (0.915 m/s).
- Handling Constraints: DCT successfully navigated scenarios where other methods failed due to treating movable obstacles as hard constraints (e.g., Case 2 with a narrow side path).
- Mixed Cluttered Environments: DCT showed significant performance gains in mixed cluttered environments, with success rates increasing and navigation costs decreasing as more obstacles were designated as movable. In highly cluttered fixed-obstacle environments (F4M0), success rates dropped, highlighting the importance of contact tolerance.
- Real-World Verification: DCT demonstrated successful navigation through a curtain and among movable boxes on a real robot, showcasing its ability to handle deformable and arbitrary-shaped obstacles.
Limitations and Future Work
- VLM Reliance: While powerful, the performance of VPP is dependent on the capabilities of the chosen VLM. The paper notes that some VLMs introduced a risk of collision due to misidentified obstacles.
- Computational Cost: Although addressed by the DNN, the underlying VLM inference can still be a bottleneck, necessitating the memory-driven approach.
- Future Work: The authors suggest extending DCT towards richer multi-modal reasoning (beyond vision and language) and scaling up to more complex, large-scale real-world deployments.
Summary updated