V2A-Vision to Action: Learning robotic arm actions based on vision and language

Michal Nazarczuk, Krystian MikolajczykOpenReview 2022 · OpenReview

TL;DR

This paper introduces Vision to Action (V2A), a new AI task that challenges agents to generate a sequence of robotic arm actions based on visual input and natural language instructions. They also present a novel multimodal attention model and a new dataset to tackle this problem, aiming to bridge the gap between visual reasoning and robotic manipulation.

What Problem Does This Paper Address?

Current AI research has made strides in visual question answering (VQA) and embodied question answering (EQA), which involve understanding visual scenes and language. However, there's a gap in tasks that require an agent to not only understand but also act upon that understanding in a physical or simulated environment. Specifically, this paper addresses the challenge of teaching a robotic arm to perform high-level tasks (like stacking objects) by interpreting natural language instructions and reasoning about a visual scene to generate a sequence of primitive actions. This requires a combination of visual perception, language understanding, and action planning, which is crucial for developing more capable and interactive robots.

Methods

The authors propose a Multimodal Attention Model for the V2A task, inspired by previous work in neural-symbolic VQA. The approach involves three main steps:

  1. Scene Segmentation: Using Mask R-CNN, the model identifies and categorizes objects within the input image.
  2. Attributes Extraction: A ResNet-34 network processes the segmented object images and the full scene to extract object properties and their spatial relationships, creating a disentangled representation of the visual scene.
  3. Instruction to Action Translation: This is framed as a sequence-to-sequence translation problem. Both the extracted visual attributes and the natural language instruction are encoded using Bidirectional LSTMs. A unidirectional LSTM decoder then generates the action sequence, utilizing a cross-modal attention mechanism to effectively fuse information from both modalities. This attention mechanism allows the model to focus on relevant parts of both the visual scene and the instruction when predicting each action.

They also train a "blind" model that only uses language input as an ablation study to highlight the importance of visual perception.

Datasets

The paper introduces a new V2A dataset built upon the existing SHOP-VRB dataset. This new dataset includes:

  • Scenes: Similar to SHOP-VRB, featuring tabletop arrangements of common household objects.
  • Instructions: Natural language instructions grounded in the visual scenes, asking the agent to perform specific tasks. These instructions are designed to require visual reasoning.
  • Engine: A novel engine is developed to assess whether a predicted sequence of primitive actions successfully completes the given task. This engine allows for the generation of ground truth action sequences and validation of model outputs.

The dataset is split into training, validation, test, and benchmark sets. The benchmark split is particularly interesting as it uses new object instances of the same classes, aiming to test the model's generalization capabilities.

Key Insights and Ideas

  • Bridging Vision, Language, and Action: The V2A task is a significant step towards agents that can not only understand but also act in the world, integrating visual reasoning with action planning.
  • Importance of Multimodal Reasoning: The authors demonstrate that visual input is crucial for successful action planning, as shown by the significantly poorer performance of the "blind" (language-only) model.
  • Challenging Nature of Complex Tasks: Tasks involving manipulation of multiple objects are considerably harder than those involving single objects, mirroring human intuition.
  • Dataset and Engine Design: The creation of a dataset with complex instructions and a robust engine for validation is critical for advancing research in this area. The engine's ability to assess task completion based on functional primitives is a key enabler.
  • Impact of Attribute Quality: The performance of the multimodal model is sensitive to the quality of the extracted visual attributes, highlighting the importance of accurate scene understanding.

Key Results

The paper evaluates models using several metrics, including Success Rate (SR), Valid Success Rate (VSR), Invalid Success Rate (ISR), Harmonic Success Rate (HSR), Efficiency Score (Escore), and Action Sequence Accuracy (Aacc).

  • The Multimodal Attention Model (MAM) significantly outperforms the Text LSTM (blind) model across most metrics. For instance, on the test split, MAM achieves a VSR of 24.9% (using ground truth attributes) compared to the Text LSTM's 3.1%.
  • The MAM model shows a noticeable drop in performance on the benchmark split compared to the test split, especially when using inferred attributes. This suggests that generalizing to new object instances, even with familiar classes, is challenging.
  • The attributes recognition accuracy (Table 5) directly correlates with the performance of the MAM model, with higher accuracy leading to better results.
  • The Escore indicates that models generally do not add many unnecessary actions, likely due to the supervised pre-training phase.
  • Aacc shows that while models can produce sequences that are executable by the engine, a smaller fraction are focused on the actual task.

Limitations and Future Work

  • Generalization to New Instances: The performance drop on the benchmark split indicates that generalizing to unseen object instances remains a challenge.
  • Complexity of Multi-Object Tasks: Tasks involving multiple objects are significantly harder, suggesting a need for more sophisticated planning strategies.
  • Attribute Quality: The reliance on accurate attribute extraction highlights the need for robust perception systems.

Future work could focus on improving generalization, developing more advanced planning algorithms for complex tasks, and enhancing the robustness of visual perception. The authors also suggest that the ground truth sequences could be used for initializing models that are then trained on other data.

Summary updated

Abstract

In this work, we present a new AI task-Vision to Action (V2A)-where an agent (robotic arm) is asked to perform a high-level task with objects (eg stacking) present in a scene. The agent has to suggest a plan consisting of primitive actions (eg simple movement, grasping) in order to successfully complete the given task. Instructions are formulated in a way that forces the agent to perform visual reasoning over the presented scene before inferring the actions. We extend the recently introduced dataset SHOP-VRB with task instructions for each scene as well as an engine capable of assessing whether the sequence of primitives leads to a successful task completion. We also propose a novel approach based on multimodal attention for this task and demonstrate its performance on the new dataset.
BibTeX
@misc{nazarczuk2022v,
  author = {Michal Nazarczuk and Krystian Mikolajczyk},
  title = {{V2A-Vision to Action: Learning robotic arm actions based on vision and language}},
  year = {2022},
  url = {https://openreview.net/forum?id=V9kTQdos1Jk},
  howpublished = {OpenReview},
}

Loading related content...

Figure