Motion Propagation via Appearance Self Similarity

This direction is merged into another direction: occlusion-induced correspondence failure (occlusion handling). Its last version stays readable and citable here.

Version 1 of 114 papers24 conceptsfrozen 2026-10-04

Optical flow and point tracking traditionally rely on a simple assumption: a pixel in Frame A must look like its corresponding pixel in Frame B. This "photometric consistency" is the engine of motion estimation, but it fails catastrophically when pixels disappear. Whether an object moves behind an obstacle, slides off-screen, or is obscured by a change in pose, occlusions create "black holes" where the data required for matching simply does not exist.

A growing body of research suggests a solution: when you cannot see where a pixel went, look at its neighbors that you can still see. By leveraging the self-similarity of an object's appearance, models can "borrow" motion from visible regions to fill in the gaps for hidden ones.

The Occlusion Problem

In unsupervised learning for optical flow, models like OccInpFlow by Kunming Luo et al. face a fundamental hurdle. Without ground truth labels, the model only knows it is right if the warped pixels match the target frame. If a pixel is occluded, there is no target to match against, and the loss function provides no guidance.

Historically, researchers tried to handle this with local smoothness constraints—assuming a pixel moves like its immediate neighbors. However, this fails at object boundaries or for large occlusions where the nearest visible neighbor might be many pixels away.

Appearance as a Proxy for Motion

The core intuition behind recent breakthroughs is that pixels that look alike usually move alike. If two pixels share similar features—texture, color, or semantic context—they likely belong to the same physical object. Since objects tend to move as rigid or semi-rigid units, their constituent pixels share a motion trajectory.

Dylan Campbell and colleagues formalized this in Learning to Estimate Hidden Motions with Global Motion Aggregation. They introduced the Global Motion Aggregation (GMA) module, which uses a transformer-style attention mechanism to identify similarities across the entire reference frame. Instead of looking only at immediate neighbors, GMA computes a "self-similarity matrix" for the whole image. If a pixel is occluded, the model identifies visible pixels with high similarity and aggregates their motion vectors to predict the hidden pixel's displacement.

From Local Matching to Global Reasoning

This shift from local search to global reasoning has redefined the state-of-the-art. In GMFlow: Learning Optical Flow via Global Matching, Haofei Xu et al. argue that optical flow should be treated as a global matching problem rather than a local regression task. By using self-attention to propagate flow information, GMFlow ensures that motion estimates in occluded areas remain consistent with the broader context of the scene.

This mechanism works because the features extracted by modern neural networks are "object-aware." Even without explicit segmentation masks, the feature similarity between two points on a car is higher than the similarity between a car and the background. Azin Jahedi and the team behind CCMR utilize this by using global context features to guide motion reasoning across multiple scales, ensuring that even high-resolution details benefit from this global consensus.

Beyond 2D Optical Flow

The principle of feature-aware propagation is now being applied to more complex spatial problems:

  • Long-term Point Tracking: In Context-PIPs, Xiaoyu Shi demonstrates that tracking a single point is harder than tracking a group. By identifying "auxiliary points" that share visual characteristics with the target, the system can maintain a robust trajectory even when the primary target point is hidden for many frames.
  • 3D Scene Flow: Yushan Zhang et al. extended this to 3D in GMSF, using a self-similarity matrix to regularize motion fields in occluded regions of a 3D point cloud.
  • Video Synthesis and Stabilization: When stabilizing video, the edges of the frame often disappear. Yufei Xu et al. showed in Out-of-boundary View Synthesis that motion vectors can be propagated into these "black holes" by estimating affinity kernels based on color and structural similarity.

Why It Works

The reason this approach is so effective is that it bypasses the need for explicit "objectness." Traditional computer vision often struggled with whether to use "top-down" (object-based) or "bottom-up" (pixel-based) methods. Motion propagation via self-similarity provides a middle ground: it uses the latent structure of the image to group pixels dynamically.

By treating the reference frame as a source of truth for "what belongs with what," these models can infer the behavior of the unseen from the behavior of the seen, turning the weakness of occlusion into a problem of relational reasoning.

Papers in this version

History & citationVersion 1 · 2026-10-04

Merged into another direction · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_motion_propagation_via_appearance_self_s_5663d733,
  title = {Motion Propagation via Appearance Self Similarity},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/motion-propagation-via-appearance-self-similarity/rdr_5663d7331296439b9b13147b20a98087},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_motion_propagation_via_appearance_self_s_5663d733_v1,
  title = {Motion Propagation via Appearance Self Similarity},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/motion-propagation-via-appearance-self-similarity/rdr_5663d7331296439b9b13147b20a98087/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}