Abstract
Traditional 3D reconstruction methods like Neural Radiance Fields (NeRF) excel at capturing surface geometry but remain blind to internal structures. We present Structure from Collision (SfC), a framework that infers the hidden interior of non-transparent objects by observing their deformation during physical interactions. By coupling a differentiable material point method (DiffMPM) with neural radiance fields, SfC-NeRF optimizes internal density distributions based on visual collision cues. Our approach demonstrates that physical motion serves as a powerful proxy for internal density, enabling the reconstruction of cavities and varying mass distributions without specialized imaging hardware.
Introduction
Current state-of-the-art 3D reconstruction technologies, such as NeRF and 3D Gaussian Splatting, are limited to capturing the "skin" of an object. For applications in robotics and high-fidelity simulation, this "surface-only" view is insufficient. For instance, a robot must distinguish between a solid and a hollow object to apply appropriate grasping force. Identifying internal structures is fundamentally ill-posed because the interior is occluded from the camera.
SfC-NeRF addresses this by using physics as a probe. By observing how an object crumples or bounces during a collision, the model can infer its internal makeup. This method treats internal structure estimation as an optimization problem constrained by the laws of physics.
Methodology
The SfC-NeRF pipeline integrates four primary components to solve the inverse problem of internal structure estimation:
- Differentiable Physics Constraints: Built upon PAC-NeRF, the system utilizes a differentiable material point method (DiffMPM). This allows the model to backpropagate errors from the observed visual appearance of a collision back to the underlying physical particles.
- Appearance-Preserving Constraints: To maintain the integrity of the known surface, the model employs loss functions that enforce the object's initial static appearance during the optimization of its interior.
- Keyframe Constraints: The optimization prioritizes frames immediately following a collision, where internal structures exert the most significant influence on external deformation.
- Volume Annealing: To navigate the complex, non-convex optimization landscape, the authors introduce a volume annealing process. By repeatedly contracting and expanding the internal volume, the model can escape local optima and converge toward the true internal structure.
The process begins by learning a "filled" object from a static image and subsequently "carves out" the internal structure by fitting the model to a video sequence of a collision.
Experimental Setup and Dataset
To evaluate this novel task, the authors introduced the SfC Dataset, comprising 115 unique objects. The dataset features five base external shapes (sphere, cube, bicone, cylinder, and diamond) with diverse internal cavities varying in shape, size, and location. Materials range from elastic solids and plasticine to fluids and granular sand. Ground truth data was generated using high-fidelity physics simulators (MLS-MPM) and photorealistic rendering.
Results and Discussion
The performance of SfC-NeRF was measured using the Chamfer Distance (CD) between estimated and actual internal particles.
- Accuracy: SfC-NeRF achieved a CD of 0.165, significantly outperforming static models (0.720) and physics-informed baselines like LPO (0.656).
- Localization: The model accurately identified the spatial bias of internal holes (e.g., left vs. right) based on the object's tilting behavior during impact.
- Predictive Power: By understanding internal mass distribution, SfC-NeRF excels at future state prediction, achieving a PSNR of 26.60 compared to 23.44 for models assuming a uniform solid interior.
Conclusion and Future Work
SfC-NeRF demonstrates that physical interaction can reveal hidden internal structures. However, the method currently requires observable deformation; rigid objects like steel bricks do not provide sufficient collision cues. Future work will focus on simultaneously estimating material properties and structure, as well as bridging the "sim-to-real" gap by applying the method to real-world video data.
Summary updated