Modern 3D reconstruction techniques like Neural Radiance Fields (NeRF) and 3D Gaussian Splatting (3DGS) have achieved breathtaking visual fidelity. However, to a computer, these models are often "hollow shells"—thin envelopes of color and density with no internal substance. If you try to slice a virtual watermelon reconstructed this way, or drop a neural vase, the illusion breaks: the interior is empty, and the object lacks the mass or structural integrity to react naturally.
A growing body of research is bridging this gap by integrating continuum mechanics and volumetric physical properties directly into neural representations. The goal is to move beyond "looking right" to "behaving right," creating simulation-ready assets that possess mass, friction, and elasticity.
The Problem of Surface-Centric Representations
Traditional 3D reconstruction focuses on the visible exterior. As noted by Fangyu Wu and Yuhao Chen in FruitNinja, most digital assets are "empty envelopes." This creates three primary failures in virtual environments:
- Structural Collapse: Without internal volume, objects cannot support their own weight during deformation.
- Inaccurate Dynamics: Simulators cannot calculate how an object should bounce or slide without knowing its mass distribution.
- Interaction Gaps: Tasks like slicing, fracturing, or crushing reveal the "hollow" nature of the model, as there is no data for the newly exposed surfaces.
To solve this, researchers are reimagining rendering primitives as physical particles.
Primitives as Physical Particles
The most significant shift in this field is the unification of rendering and simulation. In PhysGaussian, Tianyi Xie and colleagues treat 3D Gaussian kernels not just as "blobs" of color, but as physical particles governed by the laws of physics.
By embedding continuum mechanics—specifically the Material Point Method (MPM)—directly into the Gaussian Splatting pipeline, the Gaussians themselves carry physical strain and stress. This allows a model to undergo complex deformations, like a squishy toy being squeezed or bread being shattered, while maintaining visual consistency. Similarly, Simon Le Cleac’h and Jiajun Wu introduced DANO (Differentiable Physics Simulation of Dynamics-Augmented Neural Objects), which treats neural density fields as probabilistic volumes that robots can interact with, estimating mass and friction directly from video.
Inferring the Invisible Interior
If a camera only sees the surface, how can we know what is inside? Researchers are using two primary strategies: learned priors and dynamic observation.
Learned Material Manifolds
Models like VoMP (Predicting Volumetric Mechanical Property Fields), developed by Rishit Dagli and David Levin I.W., use deep learning to predict internal properties like Young’s modulus (stiffness) and density from visual features. By mapping an object's appearance to a "material manifold," these models can estimate the heterogeneous internal composition of an object in seconds, transforming a static mesh or NeRF into a solid, simulation-ready volume.
Structure from Motion and Collision
Another approach is to "see" the interior by watching how the exterior moves. In SfC-NeRF (Structure from Collision), researchers estimate internal density by observing how an object deforms when it hits the ground. Similarly, TopoGaussian (Inferring Internal Topology Structures from Visual Clues) by Xiaoyu Xiong and Tao Du uses a differentiable simulator to "reverse-engineer" hidden internal structures—like the hollow cavity inside a ball—based on its observed physical behavior in video.
Enabling Realistic Interaction
The ultimate goal of this research is to enable "open-world" interaction where objects can be manipulated in ways not seen in the training data.
- Fracture and Slicing: Fracture-GS by Xiaogang Wang uses 3D Gaussians to "heal" newly exposed surfaces during a collision, allowing objects to break apart realistically.
- Fluid Dynamics: Vid2Fluid by Alan Zhao extracts physical properties like viscosity from single-view videos, allowing captured fluids to be re-simulated and edited.
- Aerodynamics: Gaussian Swaying by Hongru Yan evolves Gaussians into "surface patches" that can interact with wind fields, enabling realistic swaying of trees or rippling of cloth.
By moving from hollow shells to volumetric, physics-informed representations, this research direction is turning 3D captures from static pictures into functional, interactive digital matter. As Hsiao-Yu Chen and Etienne Vouga demonstrate with Virtual Elastic Objects (VEOs), the future of 3D vision lies in capturing not just how the world looks, but how it feels and moves.