Point2Act: Efficient 3D Distillation of Multimodal LLMs for Zero-Shot Context-Aware Grasping

Sang Min Kim, Hyeongjun Heo, Junho Kim, Yonghyeon Lee, Young Min KimarXiv 2025 · DOI · arXiv

Abstract

Point2Act is a novel framework designed to enable robots to perform complex, context-aware grasping tasks by distilling 2D point predictions from Multimodal Large Language Models (MLLMs) into a lightweight 3D relevancy field. Unlike traditional 3D feature fields that rely on high-dimensional semantic vectors, Point2Act utilizes sparse point predictions to achieve high spatial precision and robust reasoning. The system achieves a 93% success rate in part-level grasping and operates in just 16.5 seconds, representing a significant improvement in efficiency and accuracy over existing baselines.

Introduction

For robotic systems to be truly effective in human-centric environments, they must move beyond simple object detection to understand nuanced context. Instructions such as "pick up the mug closer to the orange" or "grasp the handle of the red mug" require both high-level reasoning and precise spatial localization.

Overview of the Point2Act system performing a context-aware grasping task.

Existing approaches often utilize 3D Feature Fields (e.g., LERF or F3RM), which "lift" high-dimensional semantic features like CLIP vectors into 3D space. However, these methods face two primary challenges:

  1. Efficiency: Storing and querying 512-dimensional vectors at every point in a 3D scene is computationally expensive and memory-intensive, often leading to processing times of several minutes.
  2. Precision: Semantic maps produced by these methods tend to be "blurry," making it difficult to pinpoint exact 3D coordinates for precise grasping, particularly for part-level tasks.

Methodology

Point2Act addresses these limitations by shifting from dense feature distillation to sparse point prediction. The framework follows a multi-step pipeline:

  1. MLLM as a Point Predictor: The system prompts an MLLM (specifically Molmo) to analyze 2D images of a scene and predict a single 2D coordinate corresponding to the user's request.
  2. 3D Relevancy Fields: These 2D points are projected into 3D space to build a "relevancy field"—a 3D map where each point is assigned a scalar value (0 to 1) indicating its task relevance. This scalar representation is significantly more efficient than high-dimensional feature fields.
  3. Multi-view Aggregation: To mitigate MLLM errors or occlusions, Point2Act queries the model from multiple angles and distills these 2D "votes" into a consistent 3D field.
  4. Pipelined Execution: The authors designed a pipeline that overlaps image capture, MLLM querying, and 3D reconstruction, reducing total latency.
  5. Grasp Selection: Potential grasp candidates are generated via AnyGrasp and filtered based on their alignment with the highest-scoring regions in the 3D relevancy field.
Visualization of a 3D relevancy field for grasping flowers by their stems.

Experimental Evaluation

The system was evaluated on eight real-world tabletop scenes with 90 unique language prompts for 3D localization accuracy, and four complex scenes with 20 prompts for grasping performance. Point2Act was compared against established baselines, including LERF-TOGO, F3RM, GaussianGrasper, and GraspSplats.

Results and Discussion

Point2Act demonstrates superior performance in both speed and accuracy:

  • Speed: The entire pipeline, from initial image capture to robot movement, takes only 16.5 seconds—a 59% speedup over sequential processing methods.
  • Accuracy: In part-level grasping tasks (e.g., "grasp the handle"), Point2Act achieved a 93% success rate, significantly outperforming LERF-TOGO (45%) and F3RM (41%).
  • Robustness: The system handled "context-level" queries (identifying objects based on spatial relationships) with 98% accuracy.

The use of sparse points makes the system "token-efficient," while multi-view aggregation allows a clear "true" location to emerge from noisy MLLM predictions. Furthermore, the integration of a full MLLM enables reasoning capabilities that CLIP-based models lack, such as quantity comparison or identifying abstract affordances.

Conclusion

Point2Act provides an efficient and precise method for zero-shot context-aware grasping. While currently limited to static scenes and task-specific queries, it represents a significant step toward more capable and responsive robotic assistants. Future work will focus on enabling dynamic scene updates and more flexible human-robot interaction.

Summary updated

Abstract

We propose Point2Act, which directly retrieves the 3D action point relevant to a contextually described task, leveraging Multimodal Large Language Models (MLLMs). Foundation models opened the possibility for generalist robots that can perform a zero-shot task following natural language descriptions within an unseen environment. While the semantics obtained from large-scale image and language datasets provide contextual understanding in 2D images, the rich yet nuanced features deduce blurry 2D regions and struggle to find precise 3D locations for actions. Our proposed 3D relevancy fields bypass the high-dimensional features and instead efficiently imbue lightweight 2D point-level guidance tailored to the task-specific action. The multi-view aggregation effectively compensates for misalignments due to geometric ambiguities, such as occlusion, or semantic uncertainties inherent in the language descriptions. The output region is highly localized, reasoning fine-grained 3D spatial context that can directly transfer to an explicit position for physical action at the on-the-fly reconstruction of the scene. Our full-stack pipeline, which includes capturing, MLLM querying, 3D reconstruction, and grasp pose extraction, generates spatially grounded responses in under 20 seconds, facilitating practical manipulation tasks. Project page: https://sangminkim-99.github.io/point2act/
BibTeX
@misc{kim2025point,
  author = {Sang Min Kim and Hyeongjun Heo and Junho Kim and Yonghyeon Lee and Young Min Kim},
  title = {{Point2Act: Efficient 3D Distillation of Multimodal LLMs for Zero-Shot Context-Aware Grasping}},
  year = {2025},
  eprint = {2508.03099},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2508.03099},
}
Figure