When we observe the real world, surfaces frequently change their connectivity. A mouth opens to reveal the cavity behind it; a piece of paper tears; a drop of water splashes into independent droplets.
For traditional static 3D reconstruction frameworks, capturing these moments is impossible. To animate or track scenes over time, modern neural representations—such as Neural Radiance Fields (NeRFs) or 3D Gaussian Splatting—typically rely on a canonical space. They learn a single, static 3D template of the scene and a time-dependent deformation field (usually parameterized by a Multi-Layer Perceptron) that warps each video frame back to this template.
This approach works beautifully for smooth, elastic movements like a bending arm. However, it fails fundamentally when a scene undergoes a topological change.
The Mathematical Conflict of Continuity
The root of the problem lies in a mathematical mismatch. A standard deformation field is a continuous mapping function. By definition, a continuous function preserves spatial neighborhoods: points that are close together in the canonical space must remain close together after the transformation.
When an object undergoes a topological change, this assumption is violated:
- Splitting/Tearing: Two adjacent points on a solid surface suddenly separate and move in opposite directions.
- Merging/Contact: Two distinct surfaces come together to form a single continuous structure.
- Appearance/Disappearance: New geometry emerges from nowhere (e.g., a hand pulling an object out of a pocket).
As noted by Park et al., because continuous MLPs are biased toward smooth transformations, they cannot naturally represent the sharp step-functions required by structural breaks. When forced to model a topological discontinuity, the deformation field attempts to stretch the canonical geometry across the gap. This results in severe visual artifacts, such as floating webs of geometry, streaking, and unnatural smearing.
Canonical Space Continuous Warp Actual Tearing
[ A ][ B ] --> [ A ]=======[ B ] --> [ A ] [ B ]
(Adjacent points) (Unnatural stretching) (True Discontinuity)
How Continuity Constraints Fail Across Representations
Different 3D representations experience this fundamental limitation in distinct ways:
Primitive-Based Representations (Gaussian Splatting)
In point- or primitive-based rendering, a fixed set of elements is tracked across time. As Gao et al. observe, when surfaces separate, these primitives stretch out into elongated shapes rather than cleanly partitioning to reflect the new geometry. Furthermore, Vo highlights that frameworks relying on a fixed set of primitives cannot easily account for objects appearing or disappearing, as the underlying number of elements is constrained to be constant.
Implicit Surface Representations (SDFs and Occupancy)
For coordinate-based neural implicit surfaces, warping fields struggle to maintain geometric integrity when surfaces make or break contact. Chen et al. point out that these transitions create spatial discontinuities that traditional coordinate mapping functions cannot resolve, leading to a loss of fine geometric details.
Physical Simulations
The challenge extends beyond computer vision into computational physics. In continuum mechanics, classical models rely on partial differential equations that require continuous displacement fields to calculate derivatives. As Yu explains, these models fail entirely at material fractures where displacements become discontinuous. Similarly, Tao et al. note that grid-based simulation frameworks lack the inherent mechanisms to handle extreme structural breaks or fluid splashing without specialized, computationally expensive modifications.
Paradigms for Modeling Discontinuous Dynamics
To overcome the limitations of continuous deformation fields, researchers have developed three main strategies.
1. Lifting to Higher Dimensions (Hyper-Spaces)
Instead of trying to model the discontinuity directly in 3D space, this paradigm embeds the scene into a higher-dimensional space. As demonstrated by Park et al. in HyperNeRF, a continuous deformation field can operate smoothly in a higher dimension without tearing. The actual 3D scene is reconstructed by taking a time-varying 3D "slice" of this constant, higher-dimensional shape.
When a mouth opens, the higher-dimensional surface does not rip; instead, the slicing plane simply intersects a different part of the higher-dimensional geometry. This concept has been successfully adapted by Giebenhain et al. for dynamic head tracking and Duggal et al. to handle structural variations across object categories.
2. Eliminating the Canonical Space via Temporal Interpolation
Another approach abandons the static template entirely. If there is no canonical space to warp back to, the model is not constrained by a continuous mapping function across long time horizons.
Park et al. demonstrated that simple temporal interpolation of feature vectors within a 4D hash-grid can naturally bypass coordinate warping. Because the network interpolates features directly in space-time, objects can split, merge, or disappear without requiring explicit deformation tracking. Similarly, residual-based streaming frameworks, such as those designed by De Mello et al. and Wang et al., focus on tracking frame-to-frame localized changes rather than global transformations, mitigating the canonical template bottleneck.
3. Part-Based and Decoupled Representations
If a scene cannot be modeled as a single elastic body, it can be broken down into independent components that move separately. Li et al. address this by segmenting scenes into distinct rigid and non-rigid instances, preventing global deformation models from producing streaking artifacts when objects move past or behind one another. In the context of shape rigging, Geng et al. utilize feature-embedded "blobs" to represent articulated structures, allowing independent parts to move without being bound by a continuous global occupancy constraint.
Key Foundational Work
Fast incorporation of optical flow into active polygons (2005) — contributed 1 concept elements to this direction.
Object Metamorphosis Based on Generalized Morphology Transformation (2001) — contributed 1 concept elements to this direction.