Large Displacement Motion Estimation via Global Correspondence

102 concepts89 papers1 proposal

To understand why tracking fast-moving objects has historically been one of the most difficult challenges in computer vision, we must look at how algorithms traditionally measure motion. Classical optical flow and tracking methods rely heavily on the small-motion assumption. By assuming that pixels only move a tiny fraction of their dimensions between consecutive frames, algorithms can leverage local image gradients to find correspondences.

However, when an object moves at high velocity, this assumption completely falls apart. Understanding why it breaks down—and how modern architectures bypass it—reveals a major paradigm shift in how computers see motion.

The Tyranny of the Small-Motion Assumption

Traditional variational optical flow algorithms rely on the brightness constancy assumption: the idea that a pixel's intensity does not change as it moves from one frame to the next. To make this assumption mathematically tractable, early methods used a first-order Taylor expansion to linearize the constraint. As noted by Frédéric Huguet et al. and Carola-Bibiane Schönlieb et al., this mathematical approximation is only valid for infinitesimal, local movements.

When an object moves rapidly, its frame-to-frame displacement exceeds this narrow local neighborhood. Local gradient descent algorithms find themselves completely blind to the correct match, getting trapped in local optima because the underlying gradient information no longer points toward the true correspondence (Qifeng Chen et al.; Haofei Xu et al.). If a tracking algorithm restricts its search to a small local window to remain computationally efficient, any velocity exceeding that predefined radius causes a catastrophic tracking failure (J. Woodfill et al.; Suihanjin Yu et al.). This failure is equally severe when the motion is caused by rapid camera ego-movement; as Trung Ngo Thanh demonstrates, local search-based feature tracking quickly breaks down when the spatial gap between frames outpaces the search window.

The Failure of Coarse-to-Fine Pyramids

The traditional remedy for large displacements was the coarse-to-fine image pyramid. By downsampling images into multiple scales, a large displacement at the original resolution (say, 32 pixels) becomes a small, manageable displacement at a coarser resolution (say, 2 pixels). Algorithms could solve the motion at the top of the pyramid using standard gradient descent and propagate that solution down as an initialization for finer scales (Anurag Ranjan et al.).

[Fine Scale]    ■■■■■■■■  ---> (Object moves 32 pixels; gradient descent fails)
                   ↓
[Coarse Scale]  ■■        ---> (Object moves 2 pixels; gradient descent succeeds)

However, this hierarchical approach introduces a fundamental flaw: the small, fast-moving object problem. As Zhuoyuan Chen et al. point out, if an object is smaller than the distance it travels, it completely vanishes at coarser scales of the pyramid. A tennis ball flying across a frame becomes a single blurred pixel or disappears entirely when downsampled. Because the algorithm cannot initialize motion for something it cannot see at the top of the pyramid, it introduces severe errors that accumulate as the flow field is refined downward (Mila Nikolova et al.; Shanshan Zhao et al.).

Breaking Free via Global Correspondence

To solve large displacement motion estimation, researchers had to abandon the reliance on local gradients and transition to global correspondence matching. Instead of assuming an object is nearby and using gradients to "walk" toward it, global methods search the entire image to find the best match, treating large displacements identically to small ones.

One elegant way to achieve this is through discrete optimization over regular grids or nearest neighbor fields. As demonstrated by Ryan Kennedy et al. in their work on hierarchically-constrained optical flow, formulating motion estimation as a discrete selection problem over a large label space makes finding a massive displacement computationally equivalent to finding a tiny one. This completely bypasses the local minima that plague continuous gradient descent. Similarly, Qifeng Chen et al. showed with Full Flow that global optimization over regular grids allows the network to find matches across the entire image without requiring separate, brittle descriptor matching steps.

Another highly successful strategy is the sparse-to-dense pipeline. By first identifying robust, invariant salient points or sparse tracks across the entire image, the algorithm establishes a reliable global "anchor" for motion, which is then interpolated into a dense flow field (Andrei Zanfir et al.; Guillaume Le Moing et al.). For instance, the authors of Flow Fields leveraged a hierarchical search that prioritizes local smoothness to expand these global matches into a dense field with far fewer outliers than traditional nearest-neighbor approaches.

Modern Deep Learning and All-Pairs Correlation

Modern deep learning architectures have institutionalized this global-to-local philosophy. Early deep networks integrated descriptor matching directly into variational frameworks to handle large displacements (Philippe Weinzaepfel et al.). Later, networks like DGC-Net combined global correlation layers with coarse-to-fine strategies to handle massive geometric transformations and viewpoint shifts (Aleksei Tiulpin et al.).

The definitive modern paradigm, exemplified by RAFT (Jia Deng et al.), completely abandons image pyramids. Instead, it constructs an all-pairs correlation volume that computes the visual similarity between every single pixel in the first frame and every single pixel in the second frame.

Frame 1 (H x W) ──┐
                  ├──> All-Pairs Correlation Volume (H x W x H x W)
Frame 2 (H x W) ──┘
                  │
                  ▼
        Iterative GRU Updates ──> Sub-pixel Accurate Flow Field

By maintaining this global look-up table at full resolution, a recurrent GRU-based operator can iteratively look up features and refine the flow field. Because the correlation volume preserves all pairwise relationships across the entire image, the network can instantly recover from large displacements and track fast-moving objects without losing fine structural details.

Even for resource-constrained robotics applications where computing a massive all-pairs volume is too expensive, the core intuition remains the same: Huaizu Jiang et al. demonstrated that a "global-to-local" scheme—combining low-resolution transformer-based global matching with lightweight CNN refinement—can achieve high accuracy for large displacements while maintaining real-time frame rates on edge devices. By replacing local gradient assumptions with global lookup mechanisms, modern motion estimation treats distance not as a barrier, but simply as another coordinate to look up.

Key Foundational Work

An Algorithm for Real-Time Tracking of Non-Rigid Objects (1991) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_large_displacement_motion_estimation_d236c11f,
  title = {large displacement motion estimation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/large-displacement-motion-estimation/rdr_d236c11f430d40a3a139711950a2e75f},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_large_displacement_motion_estimation_d236c11f_v1,
  title = {large displacement motion estimation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/large-displacement-motion-estimation/rdr_d236c11f430d40a3a139711950a2e75f/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake