Modeling Non-Rigid Deformations in Dynamic 3D Scenes

1,313 concepts970 papers23 proposals

For decades, computer vision has relied on a powerful simplifying assumption to understand movement: the rigid-body assumption. In a rigid world, any motion can be elegantly described by a 6-degree-of-freedom (6-DoF) transformation matrix consisting of a 3D rotation and a 3D translation. Under this constraint, the spatial relationships between an object's internal points never change. This mathematical elegance underpins foundational techniques like Structure from Motion (SfM), Simultaneous Localization and Mapping (SLAM), and early tracking frameworks.

However, the real world is rarely perfectly rigid. As David Novotný et al. observe in Learning 3D Object Categories by Looking Around Them, traditional SfM frameworks break down when confronted with organic or articulated entities like humans or animals. When a pedestrian walks, a cyclist turns, or fabric folds, the underlying geometry bends, stretches, and deforms. Forcing a rigid mathematical constraint onto these non-rigid motions introduces severe geometric inaccuracies, structural artifacts, and tracking failures.

To build high-fidelity digital twins, autonomous driving simulations, and robust scene-understanding models, researchers are shifting away from rigid assumptions toward explicit non-rigid motion modeling.

The Failure Modes of Rigidity

When a system optimized for rigid objects encounters non-rigid motion, it typically fails in one of two ways: it either completely misinterprets the geometry, or it treats the deformable object as an anomaly to be ignored.

In autonomous driving pipelines, for example, vehicles are easily tracked using rigid 3D bounding boxes. But as Carl Lindström et al. demonstrate in Are NeRFs ready for autonomous driving? Towards closing the real-to-simulation gap, representing human actors with these same rigid boxes fails to capture the complex articulation of limbs. This limitation directly degrades the realism of sensor simulations. If a reconstruction model attempts to force a rigid transformation onto a deforming surface, the math breaks, resulting in "ghosting" artifacts or distorted depth maps, a problem highlighted by Guofeng Zhang et al. in Simultaneous multi-body stereo and segmentation.

To protect the integrity of static maps, many dynamic SLAM frameworks simply detect and segment out non-rigid entities like pedestrians. However, as Marc Pollefeys notes in Robust Dense Mapping for Large-Scale Dynamic Environments, while removing these objects keeps the static background clean, it leaves the system entirely blind to the detailed 3D structure of the most critical dynamic actors in the environment.

Why Piecewise Rigidity Falls Short

An intuitive intermediate step to solving this problem is piecewise rigid approximation. If an entire object cannot be modeled with a single rigid matrix, perhaps it can be broken down into a collection of smaller, rigid parts.

For instance, in RigidFlow: Self-Supervised Scene Flow Learning on Point Clouds by Local Rigidity Prior, Ruibo Li et al. decompose scenes into "supervoxels," assuming that each local cluster moves rigidly. While this approach works reasonably well for articulated mechanical objects (like a laptop lid or a car door), it struggles with highly fluid or elastic deformations.

Rigid Motion:      [ Point A ] ---> ( Same Distance ) ---> [ Point B ]
Piecewise Rigid:   [ Cluster 1 (Rigid) ] <--- Discontinuity ---> [ Cluster 2 (Rigid) ]
Non-Rigid Flow:    [ Point A ] -\                               /- [ Point B ]
                                 \--- Continuous Stretching ---/

As Diwen Wan et al. explain in Superpoint Gaussian Splatting for Real-Time High-Fidelity Dynamic Scene Reconstruction, clustering primitives into rigid groups fails to capture fine-grained, continuous details such as the stretching of skin or the folding of fabric. Over-segmenting an object into tiny rigid components can approximate deformation, but it introduces unnatural discontinuities at the boundaries and fails to represent the true physical nature of elastic materials.

Continuous and Learned Deformations

To truly capture non-rigid dynamics, recent architectures replace discrete coordinate transformations with continuous, non-linear representations. Instead of asking “how did this object rotate and translate?” these models ask “how did each point in space deform over time?”

1. Coordinate and Displacement Fields

Rather than relying on rigid 3D matrices, some frameworks model motion as a continuous coordinate warp. In Region Deformer Networks for Unsupervised Depth Estimation from Unconstrained Monocular Videos, Haofei Xu et al. introduce a flexible 2D displacement field driven by a bicubic deformation function. This allows the network to map pixel movements smoothly across deforming surfaces, bypassing standard geometric constraints entirely to accurately estimate depth for walking pedestrians.

2. Neural Motion Flow Fields

In 4D reconstruction (such as dynamic NeRFs or Gaussian Splatting), researchers are integrating Neural Motion Flow Fields (NMFFs). Cheng Zhao et al. leverage this in SplatFlow to track complex scene dynamics. By representing motion as a continuous vector field parameterized by a neural network, the system can represent non-linear, time-varying deformations—like human gestures or animal movements—that standard linear flow fields cannot capture.

3. Non-Linear Embedding Alignment

For unstructured data like 3D point clouds, linear interpolation between frames causes severe structural tearing when objects bend or stretch. To solve this, Yiming Zeng et al. propose IDEA-Net, which pairs a coarse linear motion estimate with a learned, non-linear refinement layer. By aligning point features in a deep embedding space rather than raw Euclidean space, the network learns point-wise correspondences that respect the underlying physical topology of the deforming object.

By moving from rigid geometric constraints to these flexible, continuous, and learned deformation fields, modern vision systems are finally beginning to reconstruct and understand the non-rigid reality of the moving world.

History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_non_rigid_motion_modeling_d0f9361b,
  title = {non-rigid motion modeling},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/non-rigid-motion-modeling/rdr_d0f9361bbc3e4fb68685f00e9ba498ac},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_non_rigid_motion_modeling_d0f9361b_v1,
  title = {non-rigid motion modeling},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/non-rigid-motion-modeling/rdr_d0f9361bbc3e4fb68685f00e9ba498ac/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake