Generalizing Physical Dynamics Across Categories and Materials

85 concepts74 papers1 proposal

When humans see a novel object—say, a strange geometric sculpture made of rubber—we can instantly predict how it will bounce, deform, or slide. We do not need to see thousands of examples of that specific sculpture category to understand its behavior. Traditional deep learning models, however, struggle with this task. When trained on raw pixels or specific object categories, they tend to memorize superficial visual textures and specific motion trajectories, failing completely when introduced to out-of-distribution materials or geometries.

To bridge this gap, a growing body of research focuses on cross-domain physical generalization. The core intuition is simple: while visual appearances vary infinitely, the underlying laws of physics—such as conservation of momentum, gravity, and contact mechanics—are invariant. By forcing models to learn these universal principles rather than category-specific patterns, agents can generalize zero-shot across entirely different materials, shapes, and environments.

Decomposing the World into Object-Centric and Particle Abstractions

Why does a model trained to predict the dynamics of a few simple blocks fail when applied to a complex scene? Standard architectures entangle the dynamics of the entire scene into a global representation. If the number of objects changes, the representation breaks.

To fix this, researchers leverage object-centric regularizers. Inspired by developmental psychology, Piloto demonstrated in Intuitive physics learning in a deep-learning model inspired by developmental psychology that decomposing visual environments into discrete, interacting objects allows models to internalize common-sense physical rules like object permanence and solidity. Because these rules are learned at the object level, they generalize seamlessly to entirely novel shapes and movement patterns.

When this object-centric view is translated into forward modeling, scenes are represented as graphs of interacting entities. For instance, Ye et al. used graph-based Interaction Networks in Object-centric Forward Modeling for Model Predictive Control to model object collisions, showing that physics learned on simple synthetic shapes transfer directly to novel real-world objects. This compositionality is incredibly powerful: the authors of Toward Compositional Generalization in Object-Oriented World Modeling showed that mastering physical rules for a small, fixed number of object slots allows a model to generalize to an infinite library of objects, provided the action-object binding is solved. Similarly, Saanum et al. showed in Simplifying Latent Dynamics with Softly State-Invariant World Models that state-invariant world models trained on complex scenes can successfully predict dynamics in simpler scenes by applying the underlying physics rather than memorizing state-dependent patterns.

For highly deformable or elasto-plastic materials like dough or mud, object boundaries blur. In these domains, particle-based representations take over. Shi et al. showed in RoboCook: Long-Horizon Elasto-Plastic Object Manipulation with Diverse Tools that dynamics models tracking the geometric evolution of particle clouds achieve zero-shot generalization to entirely novel materials with different physical properties. By focusing on local particle interactions rather than global material identities, neural simulators can scale up from simple two-body interactions to complex multi-body collisions zero-shot, as demonstrated by Chen et al. in EqCollide: Equivariant and Collision-Aware Deformable Objects Neural Simulator.

Grounding Hidden Intrinsic Dynamics

An object's motion is dictated not just by its visible shape, but by its hidden intrinsic properties: mass, friction, elasticity, and stiffness. If a model can infer these latent parameters from brief visual or physical observations, it can simulate how any geometry composed of that material will behave.

This process, often called system identification or visual grounding, allows for remarkable cross-material transfer. Cao et al. introduced a neural material adaptor in NeuMA: Neural Material Adaptor for Visual Grounding of Intrinsic Dynamics that grounds intrinsic dynamics from video, enabling the model to learn material behavior from one object and accurately simulate how a completely different shape made of the same material would bounce or collide. To achieve this cleanly, Lin et al. utilized symbolic physics models derived from visual observations in VisionLaw: Inferring Interpretable Intrinsic Dynamics from Visual Observations via Bilevel Optimization, proving that explicit physical laws offer far superior temporal and geometric generalization than black-box neural networks.

When explicit parameters are known or inferred, models can adapt their strategies dynamically. For example, Shimada et al. conditioned 3D hand-object motion synthesis explicitly on object mass in MACS: Mass Conditioned 3D Hand and Object Motion Synthesis, allowing manipulation strategies to naturally adapt from fingertip grips to full-palm grasps across unseen geometries. In robotics, Matl et al. demonstrated in Inferring the Material Properties of Granular Media for Robotic Tasks that inferring hidden particle-scale properties (like friction and restitution) from simple material piles allows a robot to accurately predict material behavior in complex, entirely different pouring and shaping configurations.

Even when these properties are completely hidden from sight, models can learn to infer them through alternative modalities or brief interactions. Rasheed et al. showed in Learning to Measure the Static Friction Coefficient in Cloth Contact that vision-based models can estimate static friction coefficients across unseen materials and camera angles by learning the fundamental relationship between motion and friction. Alternatively, Wang et al. developed a framework in AdaAfford: Learning to Adapt Manipulation Affordance for 3D Articulated Objects via Few-Shot Interactions where a robot performs a few "test-time" interactions to quickly map force responses to underlying kinematic properties, enabling generalization to completely novel object categories.

Geometry-Agnostic Interaction Mechanics

A major hurdle in physical generalization is the sheer diversity of object shapes. If an interception or manipulation model is tied to a specific object mesh, it fails when encountering a new category. The solution is to decouple the interaction mechanics from global geometry, focusing instead on localized force vectors, contact vertices, or tool interfaces.

Consider the problem of catching a moving object. Early reinforcement learning work by Das et al. in Catching a Baseball: A Reinforcement Learning Perspective Using a Neural Network showed that agents trained on a narrow range of physical trajectories can extract robust, invariant physical principles to generalize to significantly different launch angles and distances. Expanding this to visual inputs, Zeng et al. built trajectory forecasters in Visual Reaction: Learning to Play Catch With Your Drone that allow drones to catch entirely unseen categories of thrown objects by learning the fundamental dynamics of motion rather than memorizing specific object geometries.

When interacting directly with objects, modeling the physics of contact points provides a geometry-agnostic bridge. Xu et al. showed in InterDreamer: Zero-Shot Text to 3D Dynamic Human-Object Interaction that modeling human-object interactions through the motion of contact vertices on the human body allows a physics model to generalize zero-shot to completely unseen object categories. By focusing on local contact dynamics rather than global object shapes, the system applies learned physical behaviors universally. This aligns with the findings of Ehsani et al. in Use the Force, Luke! Learning to Predict Physical Forces by Simulating Effects, who used physics simulators to train models to predict contact points and force vectors from raw video, creating actionable representations that generalize well to novel objects.

This principle applies equally to tool use and robotic manipulation. Lyu et al. trained a robot to cut paper using scissors in a physics simulator in ScissorBot: Learning Generalizable Scissor Skill for Paper Cutting via Simulation, Imitation, and Sim2Real; because the policy mastered the fundamental mechanics of the tool-object interaction interface, it achieved high zero-shot generalization to materials with wildly different stiffness and thickness, such as cardboard and aluminum foil. Similarly, Hu et al. demonstrated in REBOOT: Reuse Data for Bootstrapping Efficient Real-World Dexterous Manipulation that reusing data from prior manipulation tasks accelerates the learning of new skills because fundamental physical interactions—like applying pressure to generate torque—are highly transferable across diverse object categories regardless of geometric differences.

Universal Dynamics Across Diverse Domains

The power of learning invariant physical principles extends beyond robotics into computer vision, fluid dynamics, and generative modeling. When a model internalizes the true physics of a system, its generalization capacity spans across entirely different environments.

In computer vision, traditional models often fail when encountering novel environments due to lighting or atmospheric changes. However, Sauder demonstrated in Self-Supervised Underwater Caustics Removal and Descattering via Deep Monocular SLAM that a model that explicitly learns the physical behavior of light refraction can perform zero-shot underwater caustics removal across entirely different geological substrates and environments. In generative video modeling, Winter et al. showed in ObjectDrop: Bootstrapping Counterfactuals for Photorealistic Object Removal and Insertion that training diffusion models on counterfactual image pairs allows them to master the underlying physics of light and shadow, generalizing effectively from simple household items to large-scale outdoor structures.

Even in complex fluid dynamics and differential equations, physical invariants hold the key to generalization. Heimbach et al. trained reinforcement learning agents to act as closure models for underresolved partial differential equations in Reinforcement Learning Closures for Underresolved Partial Differential Equations using Synthetic Data. By training exclusively on inhomogeneous systems with external forcing, the learned mechanisms captured fundamental physical principles so deeply that they successfully corrected homogeneous systems without forcing zero-shot.

Ultimately, cross-domain physical generalization works because it shifts the learning objective from what an object looks like to how the physical world operates. Whether tracking general objects in the wild (InterTracker), predicting battery degradation across diverse chemistries (DITING), or navigating high-speed off-road vehicles across unstructured terrains (Biswas et al.), the most robust models are those that treat geometry and texture as temporary variables, and the laws of physics as the ultimate constants.

Key Foundational Work

Catching a Baseball: A Reinforcement Learning Perspective Using a Neural Network (1994) — contributed 1 concept elements to this direction.
Learning about Momentum Conservation (1985) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_cross_domain_physical_generalization_cro_a078b851,
  title = {cross-domain physical generalization (cross-category physical generalization)},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/cross-domain-physical-generalization-cross-category-physical-generalization/rdr_a078b851d3c545f98121813c533cd68b},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_cross_domain_physical_generalization_cro_a078b851_v1,
  title = {cross-domain physical generalization (cross-category physical generalization)},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/cross-domain-physical-generalization-cross-category-physical-generalization/rdr_a078b851d3c545f98121813c533cd68b/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake