Zero-Shot Physical Property Inference with Language Models

45 concepts33 papers1 proposal

When humans look at a scene, we do not just see shapes and colors; we perceive physical potential. We instantly recognize that a ceramic mug is heavy and fragile, whereas a paper cup is light and deformable. For computer vision and robotics, however, translating pixels into the underlying laws of physics has long been a daunting challenge. Fundamental properties like mass, friction, elasticity, and thermal diffusivity cannot be directly measured from a static 2D image or a 3D point cloud.

Traditionally, bridging this gap required manual annotation or complex, data-heavy system identification pipelines. However, a powerful new paradigm has emerged: leveraging the latent "world knowledge" embedded within Large Language Models (LLMs) and Vision-Language Models (VLMs) to map semantic labels and visual context directly to explicit physical properties.

Why It Works: Distributional Semantics as a Physics Prior

Why should a model trained primarily on text understand how a physical object bounces, slides, or absorbs heat? The answer lies in distributional semantics. Human language is rich with descriptions of physical interactions, material constraints, and common-sense affordances. Text corpora implicitly encode the relationships between objects and their physical constants—for instance, text frequently associates "steel" with high rigidity and "rubber" with high elasticity.

As Su Wang et al. demonstrated in Modeling Semantic Plausibility by Injecting World Knowledge, grounding language models in explicit physical dimensions like size, weight, and rigidity allows them to evaluate what is physically possible in the real world. This internal web of associations allows LLMs to act as zero-shot heuristic engines. By querying a model with a semantic category (e.g., "cardboard box"), it can retrieve or infer highly plausible numerical ranges for physical attributes like mass, density, or friction without any direct physical measurement, as shown by Albert J. Zhai et al. in Physical Property Understanding from Language-Embedded Feature Fields.

[ Visual Input / Mesh ] ──> [ Semantic Identification ] ──> [ LLM / VLM Knowledge Base ] ──> [ Numerical Constants ] ──> [ Physics Engine ]
     (Static Pixels)             ("Wooden Chair")             (Distributional Semantics)       (Mass, Friction, Elasticity)    (Realistic Dynamics)

From Pixels to Simulation Parameters

The primary mechanism of this research direction is translating high-level visual or semantic recognition into low-level physical constants required by deterministic physics simulators. This bridges the gap between what a camera sees and what a physics engine needs to calculate motion.

Consider the pipeline of PhysGen3D: Crafting a Miniature Interactive World from a Single Image by Boyuan Chen et al. To turn a static image into an interactive 3D environment, the system uses an LLM to estimate numerical parameters like elasticity and friction based on the identified semantic labels of objects. These parameters are then fed directly into a Material Point Method (MPM) simulator. Similarly, Shanyan Guan et al. introduced NeoWorld: Neural Simulation of Explorable Virtual Worlds via Progressive 3D Unfolding, where an LLM acts as an intermediary that translates natural language descriptions into precise simulation parameters like Young's modulus and Poisson's ratio.

By automating this mapping, researchers can bypass the "human bottleneck" of manual physics tuning. This enables the scalable creation of simulation-ready 3D asset libraries from unstructured data, as seen in urban simulation frameworks like UrbanVerse by Mingxuan Liu et al. and robotic training pipelines like Gen2Sim by Pushkal Katara et al.

Grounding Multimodal Models for Robotics

In robotics, knowing what an object is matters less than knowing how it behaves under force. Vision-Language Models (VLMs) extend the semantic-to-physical paradigm by directly processing visual data to infer physical affordances, allowing robots to interact safely with unfamiliar environments.

In Physically Grounded Vision-Language Models for Robotic Manipulation, Bidipta Sarkar et al. showed that visual features and semantic identity can serve as effective proxies for latent physical properties like fragility or mass. When a robot needs to grasp an object, it can query a VLM to predict these unobservable traits. For example, PhyGrasp by Dingkun Guo et al. combines 3D point clouds with language models to infuse robotic grasping with physical common sense, allowing a robot to adjust its grip based on textual descriptions of material fragility.

Furthermore, this physical understanding can be distilled into advanced 3D representations like Gaussian Splatting. In PUGS: Zero-shot Physical Understanding with Gaussian Splatting, Yinghao Shuai et al. queried VLMs about identified materials to distill density and stiffness estimates directly into a 3D representation, facilitating zero-shot robotic grasping. Haoyu Zhao et al. leveraged a similar approach in High-Fidelity Simulated Data Generation for Real-World Zero-Shot Robotic Manipulation Learning with Gaussian Splatting, using multimodal LLMs to automatically infer kinematic structures and physical properties from visual data to initialize stable, interactive simulations.

Beyond Mechanics: Acoustics, Optics, and Thermodynamics

The semantic-to-physical mapping paradigm is not limited to mechanical properties like mass and friction; it generalizes across diverse physical domains, demonstrating the breadth of world knowledge contained in these models:

By treating language models as repository frameworks for physical constants, modern AI bridges the gap between semantic understanding and physical reality, unlocking scalable, training-free simulation and interaction.

Key Foundational Work

Immersive audio-visual scene reproduction using semantic scene reconstruction from 360 cameras (2022) — contributed 1 concept elements to this direction.
Reconstructing Action-Conditioned Human-Object Interactions Using Commonsense Knowledge Priors (2022) — contributed 1 concept elements to this direction.
Modeling Semantic Plausibility by Injecting World Knowledge (2018) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_semantic_to_physical_property_mapping_6e1f38ae,
  title = {semantic-to-physical property mapping},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/semantic-to-physical-property-mapping/rdr_6e1f38ae8c2043f6a1e6ba6d3db479d0},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-22},
}

This exact version (version 1, frozen text):

@misc{lacuna_semantic_to_physical_property_mapping_6e1f38ae_v1,
  title = {semantic-to-physical property mapping},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/semantic-to-physical-property-mapping/rdr_6e1f38ae8c2043f6a1e6ba6d3db479d0/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-22},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake