Abstract
Point2Act is a novel framework designed to enable robots to perform complex, context-aware grasping tasks by distilling 2D point predictions from Multimodal Large Language Models (MLLMs) into a lightweight 3D relevancy field. Unlike traditional 3D feature fields that rely on high-dimensional semantic vectors, Point2Act utilizes sparse point predictions to achieve high spatial precision and robust reasoning. The system achieves a 93% success rate in part-level grasping and operates in just 16.5 seconds, representing a significant improvement in efficiency and accuracy over existing baselines.
Introduction
For robotic systems to be truly effective in human-centric environments, they must move beyond simple object detection to understand nuanced context. Instructions such as "pick up the mug closer to the orange" or "grasp the handle of the red mug" require both high-level reasoning and precise spatial localization.
Existing approaches often utilize 3D Feature Fields (e.g., LERF or F3RM), which "lift" high-dimensional semantic features like CLIP vectors into 3D space. However, these methods face two primary challenges:
- Efficiency: Storing and querying 512-dimensional vectors at every point in a 3D scene is computationally expensive and memory-intensive, often leading to processing times of several minutes.
- Precision: Semantic maps produced by these methods tend to be "blurry," making it difficult to pinpoint exact 3D coordinates for precise grasping, particularly for part-level tasks.
Methodology
Point2Act addresses these limitations by shifting from dense feature distillation to sparse point prediction. The framework follows a multi-step pipeline:
- MLLM as a Point Predictor: The system prompts an MLLM (specifically Molmo) to analyze 2D images of a scene and predict a single 2D coordinate corresponding to the user's request.
- 3D Relevancy Fields: These 2D points are projected into 3D space to build a "relevancy field"—a 3D map where each point is assigned a scalar value (0 to 1) indicating its task relevance. This scalar representation is significantly more efficient than high-dimensional feature fields.
- Multi-view Aggregation: To mitigate MLLM errors or occlusions, Point2Act queries the model from multiple angles and distills these 2D "votes" into a consistent 3D field.
- Pipelined Execution: The authors designed a pipeline that overlaps image capture, MLLM querying, and 3D reconstruction, reducing total latency.
- Grasp Selection: Potential grasp candidates are generated via AnyGrasp and filtered based on their alignment with the highest-scoring regions in the 3D relevancy field.
Experimental Evaluation
The system was evaluated on eight real-world tabletop scenes with 90 unique language prompts for 3D localization accuracy, and four complex scenes with 20 prompts for grasping performance. Point2Act was compared against established baselines, including LERF-TOGO, F3RM, GaussianGrasper, and GraspSplats.
Results and Discussion
Point2Act demonstrates superior performance in both speed and accuracy:
- Speed: The entire pipeline, from initial image capture to robot movement, takes only 16.5 seconds—a 59% speedup over sequential processing methods.
- Accuracy: In part-level grasping tasks (e.g., "grasp the handle"), Point2Act achieved a 93% success rate, significantly outperforming LERF-TOGO (45%) and F3RM (41%).
- Robustness: The system handled "context-level" queries (identifying objects based on spatial relationships) with 98% accuracy.
The use of sparse points makes the system "token-efficient," while multi-view aggregation allows a clear "true" location to emerge from noisy MLLM predictions. Furthermore, the integration of a full MLLM enables reasoning capabilities that CLIP-based models lack, such as quantity comparison or identifying abstract affordances.
Conclusion
Point2Act provides an efficient and precise method for zero-shot context-aware grasping. While currently limited to static scenes and task-specific queries, it represents a significant step toward more capable and responsive robotic assistants. Future work will focus on enabling dynamic scene updates and more flexible human-robot interaction.
Summary updated