Hierarchical Coarse to Fine Motion Alignment

You are reading version 1 of this direction, frozen 2026-10-04. The latest version is version 2.

Version 1 of 2337 papers410 conceptsfrozen 2026-10-04

In computer vision, tracking motion is a fundamental challenge. Whether we are interpolating frames in an animation, compressing a video, or estimating the flow of a glacier, we must answer a simple question: where did this pixel go?

The difficulty lies in the scale of movement. Standard convolutional neural networks (CNNs) have a limited "receptive field"—they can only see a small neighborhood of pixels at a time. If an object moves 50 pixels between frames but the network's filters only span 3x3 pixels, the network effectively "blindly" searches for a match it cannot see. To solve this, researchers have converged on a shared strategy: Hierarchical Coarse-to-Fine Alignment.

The Receptive Field Bottleneck

Imagine looking through a narrow straw at a white wall with a single black dot. If the dot moves slightly, you can follow it. If it leaps across the room, you lose it instantly. This is the "aperture problem" in motion estimation.

Traditional methods like the Lucas-Kanade algorithm, as noted by Hayato Itoh and colleagues in Local Affine Optical Flow Computation, struggle when displacements exceed the range of local linearization. In modern deep learning, even large networks face this; simply increasing depth to expand the receptive field is computationally expensive and often fails to capture the precise correspondence needed for high-quality restoration.

The Pyramid Solution

The core intuition of the hierarchical approach is to change the scale of the problem rather than the size of the filter. By building an image pyramid—a stack of the same image at progressively lower resolutions—we can turn large, "invisible" motions into small, manageable ones.

At a coarse resolution (e.g., 1/8th the original size), a 64-pixel jump becomes a mere 8-pixel shift. As Azin Jahedi demonstrates in MS-RAFT+, estimating motion at these coarse levels provides a "global" initialization. This coarse estimate acts as a rough map, telling the network generally where the object went before it attempts to find the exact pixel-perfect match at higher resolutions.

Warping and Iterative Refinement

The magic of this direction isn't just in looking at different scales, but in how those scales communicate. The standard mechanism is warping.

  1. Estimate Coarse Flow: The network predicts motion at a low resolution.
  2. Warp: This coarse flow is used to "un-move" the second frame, shifting it back toward the first frame.
  3. Refine Residuals: The network then looks at the difference (the residual) between the first frame and the warped second frame. Because the warping already handled the big jump, the remaining motion is tiny—well within the reach of standard convolutional filters.

This iterative process is central to EDVR, the winning solution of the NTIRE 2019 challenge led by Radu Timofte and Seungjun Nah. EDVR uses deformable convolutions to perform this alignment across a feature pyramid, allowing it to restore sharp details in videos with complex, dynamic blur.

Beyond Pixels: Latents and Point Clouds

While the idea originated in 2D image processing, it has proven remarkably versatile across different data structures:

  • Video Compression: In Coarse-To-Fine Deep Video Coding, Zhihao Hu and Guo Lu use a two-stage refinement to handle complex motion, allowing neural codecs to rival traditional standards like H.265.
  • 3D Point Clouds: For LiDAR data, RMS-FlowNet++ by Ramy Battrawy applies the same logic to 3D space, warping point sets at multiple scales to estimate "scene flow" for autonomous driving.
  • Generative Modeling: Even in video generation, DynamicScaler uses hierarchical guidance—generating a low-resolution "coarse" video first—to ensure that motion remains globally consistent across massive panoramic scenes.

Why It Works: Stability and Efficiency

The hierarchical strategy works because it decomposes a non-convex, high-dimensional search (finding a match anywhere in a large image) into a series of simpler, local searches.

By starting coarse, the model avoids getting stuck in "local minima"—small details that look similar but are in the wrong place. As Jipeng Lyu shows in SCas4D, optimizing large structural clusters before individual components provides a stable initialization that can speed up training by 20x.

Ultimately, hierarchical coarse-to-fine alignment is a "divide and conquer" algorithm for motion. It acknowledges that to see the trees (fine details), you must first understand the shape of the forest (global motion).

Papers in this version

Showing 60 of 337 papers.

History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 326 papers
  2. Version 1 · 2026-10-04 · 337 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_hierarchical_motion_estimation_4a3ce025,
  title = {hierarchical motion estimation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/hierarchical-motion-estimation/rdr_4a3ce025e15b4651b3236a0e6e04198d},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-21},
}

This exact version (version 1, frozen text):

@misc{lacuna_hierarchical_coarse_to_fine_motion_align_4a3ce025_v1,
  title = {Hierarchical Coarse to Fine Motion Alignment},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/hierarchical-coarse-to-fine-motion-alignment/rdr_4a3ce025e15b4651b3236a0e6e04198d/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-21},
}