NaturalVLM: Leveraging Fine-grained Natural Language for Affordance-Guided Visual Manipulation

Ran Xu, Yan Shen, Xiaoqi Li, Ruihai Wu, Hao Dong

Abstract

Enabling home-assistant robots to perceive and manipulate a diverse range of 3D objects based on human language instructions is a pivotal challenge. Prior research has predominantly focused on simplistic and task-oriented instructions, i.e., "Slide the top drawer open". However, many real-world tasks demand intricate multi-step reasoning, and without human instructions, these will become extremely difficult for robot manipulation. To address these challenges, we introduce a comprehensive benchmark, NrVLM, comprising 15 distinct manipulation tasks, containing over 4500 episodes meticulously annotated with fine-grained language instructions. We split the long-term task process into several steps, with each step having a natural language instruction. Moreover, we propose a novel learning framework that completes the manipulation task step-by-step according to the fine-grained instructions. Specifically, we first identify the instruction to execute, taking into account visual observations and the end-effector's current state. Subsequently, our approach facilitates explicit learning through action-prompts and perception-prompts to promote manipulation-aware cross-modality alignment. Leveraging both visual observations and linguistic guidance, our model outputs a sequence of actionable predictions for manipulation, including contact points and end-effector poses. We evaluate our method and baselines using the proposed benchmark NrVLM. The experimental results demonstrate the effectiveness of our approach.

Introduction

Most current research in Vision-Language Manipulation (VLM) focuses on high-level, task-oriented commands like "Slide the top drawer open." While these are intuitive for humans, they are difficult for robots to execute because they lack "how-to" details. A single high-level command often hides a complex sequence of sub-tasks, such as reaching for a handle, grasping firmly, and pulling horizontally. Existing attempts to use Large Language Models (LLMs) for generating low-level steps often fail because LLMs are not grounded in the specific physical constraints of a robot's end-effector or the nuances of a 3D scene. NaturalVLM addresses this gap by pairing diverse, human-annotated, fine-grained instructions with successful manipulation trajectories.

Methodology

The NaturalVLM framework operates in three main stages to bridge the gap between language and physical action:

  1. Instruction Selection (InstrSel): The robot identifies the current step to execute from a sequence of fine-grained instructions. This uses CLIP for text encoding and PointNet++ for visual scene features, conditioned on the end-effector's state.
  2. Affordance-Guided Manipulation:
    • AFF-NET (Affordance Network): Predicts a heatmap on the 3D point cloud indicating the interaction point (e.g., a handle).
  3. ACT-NET (Actor Network): Predicts the 6-DoF pose, gripper state, and collision avoidance parameters.
  4. Prompt Modules: To improve cross-modal alignment, the authors introduce specialized "prompt bases":
  5. Perception-Prompts: Align nouns (e.g., "handle") with visual features.
  6. Action-Prompts: Align verbs (e.g., "pull") with motion features.
Figure 5: Fine-grained instruction decomposition and cross-task alignment of actions (verbs) and objects (nouns).

The NrVLM Benchmark

The paper introduces the NrVLM benchmark, featuring 15 distinct manipulation tasks (e.g., opening bottles, turning taps, closing laptops). It contains 4,500 episodes with 82 variations using object instances from the PartNet-Mobility dataset. Crucially, NrVLM was annotated by at least 10 different humans per task to ensure linguistic variety, moving beyond templated or LLM-generated text.

Figure 2: A robot executing a complex 'pull' action in the Jenga task, guided by fine-grained instructions.

Results and Discussion

NaturalVLM was evaluated against strong baselines like PERACT and BC-Z. The framework achieved an average success rate of 0.80 on training tasks and 0.46 on novel tasks, significantly outperforming PERACT's 0.38 on novel tasks.

Key insights include:

  • Fine-grained instructions are critical: For complex tasks, high-level instructions often lead to zero success, whereas step-by-step roadmaps enable completion.
  • Decoupling Verbs and Nouns: By splitting prompt modules into "Perception" (nouns) and "Action" (verbs), the model learns generalized concepts (e.g., "grasp" applies to both handles and lids).
  • Human Data Superiority: Testing state-of-the-art LLMs (MiniGPT-4, Video-LLaMA) for instruction generation resulted in poor performance, highlighting the continued necessity of human-annotated data for high-precision robotics.

Conclusion

NaturalVLM demonstrates that fine-grained language guidance is a powerful tool for robotic manipulation. While currently limited to simulation, future work will focus on bridging the sim-to-real gap and integrating autonomous high-level planners to generate these instructions from goal-oriented commands.

Summary updated

Abstract

Enabling home-assistant robots to perceive and manipulate a diverse range of 3D objects based on human language instructions is a pivotal challenge. Prior research has predominantly focused on simplistic and task-oriented instructions, i.e., "Slide the top drawer open". However, many real-world tasks demand intricate multi-step reasoning, and without human instructions, these will become extremely difficult for robot manipulation. To address these challenges, we introduce a comprehensive benchmark, NrVLM, comprising 15 distinct manipulation tasks, containing over 4500 episodes meticulously annotated with fine-grained language instructions. We split the long-term task process into several steps, with each step having a natural language instruction. Moreover, we propose a novel learning framework that completes the manipulation task step-by-step according to the fine-grained instructions. Specifically, we first identify the instruction to execute, taking into account visual observations and the end-effector's current state. Subsequently, our approach facilitates explicit learning through action-prompts and perception-prompts to promote manipulation-aware cross-modality alignment. Leveraging both visual observations and linguistic guidance, our model outputs a sequence of actionable predictions for manipulation, including contact points and end-effector poses. We evaluate our method and baselines using the proposed benchmark NrVLM. The experimental results demonstrate the effectiveness of our approach. For additional details, please refer to https://sites.google.com/view/naturalvlm.
BibTeX
@article{xu2024naturalvlm,
  author = {Ran Xu and Yan Shen and Xiaoqi Li and Ruihai Wu and Hao Dong},
  title = {{NaturalVLM: Leveraging Fine-grained Natural Language for Affordance-Guided Visual Manipulation}},
  journal = {IEEE Robotics Autom. Lett.},
  year = {2024},
  url = {https://openreview.net/forum?id=I1FkQUXYuC},
}
Figure