Tracking Non-Rigid Objects through Complex Deformations

This direction is split into other directions: deformable object tracking. Its last version stays readable and citable here.

Version 1 of 158 papers61 conceptsfrozen 2026-10-04

Most computer vision trackers operate on a fundamental assumption: the object being followed is a relatively stable, rigid entity. We represent cars and pedestrians as bounding boxes because their proportions don't change much from frame to frame. However, this assumption collapses when faced with the "shape-shifters" of the natural world—a gymnast contorting mid-air, a growing plant leaf, or a cloud thinning in the wind.

Research in non-rigid object tracking seeks to move beyond the rigid box, developing mechanisms that can maintain an object's identity even as its silhouette, aspect ratio, and topology undergo drastic transformations.

The Failure of Rigid Templates

Traditional tracking often relies on template matching—finding a patch in frame $B$ that looks like the patch from frame $A$. As Stefan Duffner notes in PixelTrack, these global appearance models fail when geometry is fluid. If a person moves from standing to crouching, a fixed-size bounding box will either include too much background or lose parts of the target.

This "drift" occurs because the tracker cannot distinguish between the object's actual movement and its internal deformation. Anfeng He and colleagues addressed this in Siam-BM by introducing selective spatial masking and angle estimation, allowing the network to adjust to dynamic aspect ratio changes that would otherwise confuse a standard Siamese tracker.

From Bounding Boxes to Pixel Masks

To track something that changes shape, you must first understand exactly where its boundaries are. This has led to the convergence of tracking and Video Object Segmentation (VOS). Instead of a box, the tracker maintains a pixel-level mask.

Alan Lukezic demonstrated in D3S that combining a rigid motion model with a geometrically invariant pixel-similarity model allows a system to "lock on" to a target's essence while remaining flexible to its form. Similarly, MHP-VOS by Daizong Liu et al. prioritizes mask propagation over simple appearance matching. By using optical flow to "warp" the mask from one frame to the next, the system can track non-rigid deformations that would break color-based algorithms.

Modeling the Trajectory of Change

If an object’s shape is constantly changing, what stays the same? Researchers have proposed several "anchors" for identity:

  1. Edge and Contour Dynamics: Early work by Daniel P. Huttenlocher in Tracking non-rigid objects in complex scenes used the Hausdorff distance to compare edge maps. This allowed the system to track walking people by looking for "best-fit" edge alignments rather than perfect template matches.
  2. Point-Set Registration: When tracking biological entities like the nervous system of a Hydra, Thibault Lagache used Coherent Point Drift (CPD). This treats the object as a set of points that move elastically; even if the animal stretches significantly, the relative "neighborhood" of the points remains consistent enough to prevent identity loss.
  3. Deformation Trajectories: Modern Transformer-based approaches, such as the SODA framework by Lihao Liu et al., model the temporal "pathway" of a deformation. By integrating spatio-temporal cues, the system learns how a boundary (like a shadow) typically warps, allowing it to predict the next state of a non-rigid shape.

Part-Based and Hierarchical Representations

A powerful intuition in this field is that while the whole object is non-rigid, its parts might be more stable. Mingzhen Huang and the creators of DETracker suggest tracking objects as a collection of flexible "patches." This disentangles the chaotic movement of a deformable object (like a dog) from the camera's own movement.

In 3D space, Tao Yu and Yebin Liu took this further with a hierarchical approach for 3D Self-Portraits. They used a learned "inner shape prior" (the human body) to guide the deformation of an "outer mesh" (loose clothing). This dual-layer strategy allows the system to adapt to complex geometries that change rapidly as a person moves.

Handling Topological Shifts

The most extreme version of non-rigid tracking involves objects that don't just change shape, but change topology—merging or splitting apart. Sheikh Md Shakeel Hassan and the BubbleML team highlight this in fluid dynamics. Tracking bubbles in boiling liquid requires optical flow architectures specifically trained to handle the moment one object becomes two. This represents the current frontier: maintaining semantic consistency (Chengkai Hou, Key-Grid) even when the physical structure of the object is entirely redefined.

Papers in this version

History & citationVersion 1 · 2026-10-04

Split into other directions · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_tracking_non_rigid_objects_through_compl_38b1197b,
  title = {Tracking Non-Rigid Objects through Complex Deformations},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/tracking-non-rigid-objects-through-complex-deformations/rdr_38b1197b97274e4ca9976ce5b2d9d685},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_tracking_non_rigid_objects_through_compl_38b1197b_v1,
  title = {Tracking Non-Rigid Objects through Complex Deformations},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/tracking-non-rigid-objects-through-complex-deformations/rdr_38b1197b97274e4ca9976ce5b2d9d685/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}