Physically Grounded Vision-Language Models for Robotic Manipulation

Jensen Gao, Bidipta Sarkar, Fei Xia, Ted Xiao, Jiajun Wu, Brian Ichter, Anirudha Majumdar, Dorsa Sadigh

Abstract

Recent advances in vision-language models (VLMs) have led to improved performance on tasks such as visual question answering and image captioning. Consequently, these models are now well-positioned to reason about the physical world, particularly within domains such as robotic manipulation. However, current VLMs are limited in their understanding of the physical concepts (e.g., material, fragility) of common objects, which restricts their usefulness for robotic manipulation tasks that involve interaction and physical reasoning about such objects. To address this limitation, we propose PhysObjects, an object-centric dataset of 39.6K crowd-sourced and 417K automated physical concept annotations of common household objects. We demonstrate that fine-tuning a VLM on PhysObjects improves its understanding of physical object concepts, including generalization to held-out concepts, by capturing human priors of these concepts from visual appearance. We incorporate this physically grounded VLM in an interactive framework with a large language model-based robotic planner, and show improved planning performance on tasks that require reasoning about physical object concepts, compared to baselines that do not leverage physically grounded VLMs. We additionally illustrate the benefits of our physically grounded VLM on a real robot, where it improves task success rates.

Introduction

While modern Vision-Language Models (VLMs) are excellent at identifying objects or describing scenes, they often lack "physical common sense." For a robot, knowing an object is a "cup" is insufficient for safe and effective interaction; it must also understand if that cup is made of glass (fragile) or plastic (sturdy), and whether it is heavy or light.

Current VLMs struggle with these concepts because internet-scale training data rarely includes explicit labels for physical affordances. Traditional robotics approaches often rely on interaction data—the robot physically touching objects—to learn physics, but this is notoriously difficult to scale. This research explores whether we can instead leverage visual priors: teaching a model to infer physical properties from static images, much like a human knows a sponge is squishy just by looking at it.

The PhysObjects Dataset

The core contribution of this work is the PhysObjects dataset, designed to bridge the gap between visual appearance and physical reasoning. Built upon the EgoObjects dataset (egocentric videos of household objects), it contains 456K physical concept annotations.

Example of an object-centric annotation from the PhysObjects dataset, where physical properties are inferred from visual appearance.

The dataset covers eight primary concepts: Mass, Fragility, Deformability, Material, Transparency, Contents, Can Contain Liquid, and Is Sealed. It includes 39.6K crowd-sourced human annotations and 417K automated annotations. To test the model's ability to learn a broader physical representation, the researchers also included "held-out" concepts like Density and Liquid Capacity, which were used only during testing.

Methodology: Physical Grounding and Interactive Planning

The framework "physically grounds" a VLM (specifically InstructBLIP) using a two-pronged approach:

  1. Preference Learning for Continuous Concepts: For relative properties like mass or fragility, the authors use a Bradley-Terry model to train the VLM on preference pairs (e.g., "Object A is more fragile than Object B"). This allows the model to rank objects based on physical attributes rather than assigning arbitrary numerical values.
  2. Interactive Planning (Socratic Models): The fine-tuned VLM is integrated into a robotic planning loop. A Large Language Model (GPT-4) acts as the high-level planner. When faced with a physical constraint (e.g., "pick up the heaviest container"), the LLM queries the physically grounded VLM to identify the correct object in the scene before generating a sequence of robotic primitives.
Overview of the PhysObjects framework: (a) fine-tuning a VLM on physical concepts, (b) the interactive planning loop between an LLM and the grounded VLM, and (c) real-world robot execution.

Key Results and Insights

The experiments yielded several significant findings regarding the efficacy of visual physical grounding:

  • Improved Accuracy: The fine-tuned model (PG-InstructBLIP) achieved an average test accuracy of 87.5% across physical concepts, a substantial improvement over the 66.2% achieved by the base InstructBLIP.
  • Zero-Shot Generalization: On held-out concepts like Density, the grounded model outperformed the base model by 11.9%, suggesting it learned a generalizable physical representation rather than just memorizing labels.
  • Robotic Performance: In 51 real-world scenarios, the grounded planning system achieved an 88.2% success rate, compared to 56.9% for the baseline. On a Franka Emika Panda robot, the system achieved a 9/10 success rate on complex tasks requiring physical reasoning, more than doubling the performance of the non-grounded baseline (4/10).
  • Visual Appearance as a Proxy: The results confirm that semantic knowledge and visual texture are powerful enough for a VLM to infer physical properties without needing direct physical interaction.

Conclusion

By fine-tuning VLMs on the PhysObjects dataset, this work demonstrates that robots can leverage visual priors to understand the hidden physical properties of everyday objects. While the current model focuses on relative properties rather than absolute SI units, the synergy between LLMs and physically grounded VLMs provides a robust path forward for complex, multi-step robotic manipulation tasks. Future work may expand this to include 3D geometric reasoning and social reasoning to further integrate robots into human environments.

Summary updated

Abstract

Recent advances in vision-language models (VLMs) have led to improved performance on tasks such as visual question answering and image captioning. Consequently, these models are now well-positioned to reason about the physical world, particularly within domains such as robotic manipulation. However, current VLMs are limited in their understanding of the physical concepts (e.g., material, fragility) of common objects, which restricts their usefulness for robotic manipulation tasks that involve interaction and physical reasoning about such objects. To address this limitation, we propose PhysObjects, an object-centric dataset of 39.6K crowd-sourced and 417K automated physical concept annotations of common household objects. We demonstrate that fine-tuning a VLM on PhysObjects improves its understanding of physical object concepts, including generalization to held-out concepts, by capturing human priors of these concepts from visual appearance. We incorporate this physically grounded VLM in an interactive framework with a large language model-based robotic planner, and show improved planning performance on tasks that require reasoning about physical object concepts, compared to baselines that do not leverage physically grounded VLMs. We additionally illustrate the benefits of our physically grounded VLM on a real robot, where it improves task success rates. We release our dataset and provide further details and visualizations of our results at https://iliad.stanford.edu/pg-vlm/.
BibTeX
@inproceedings{gao2023physically,
  author = {Jensen Gao and Bidipta Sarkar and Fei Xia and Ted Xiao and Jiajun Wu and Brian Ichter and Anirudha Majumdar and Dorsa Sadigh},
  title = {{Physically Grounded Vision-Language Models for Robotic Manipulation}},
  booktitle = {IEEE International Conference on Robotics and Automation},
  year = {2023},
  doi = {10.1109/icra57147.2024.10610090},
}
Figure