Receptive Field Scaling for Large Motion Displacement

28 concepts25 papers1 proposal

When a video or image sequence captures fast-moving objects, standard deep learning architectures face a fundamental geometric challenge. Most computer vision models operate on the assumption of locality: they use local convolutional kernels, restricted attention windows, or localized search neighborhoods to track and align features from one frame to the next.

This assumption breaks down entirely when an object's velocity causes it to travel hundreds of pixels between consecutive frames. When a feature completely exits a network’s local processing window, the model becomes "blind" to its trajectory. It can no longer compute the gradients necessary to establish correspondences, leading to severe visual artifacts such as ghosting, blurring, or complete tracking failure.

Overcoming this mismatch between local receptive fields and large spatial displacements has become a core focus across video frame interpolation, super-resolution, deblurring, and compression.

The Mechanics of Local Blindness

To understand why traditional alignment mechanisms fail under large motion, consider how a standard convolutional or kernel-based network processes a video.

In video object detection, Kai Kang et al. observed that high-velocity objects that exit a neural network's receptive field within a few frames cannot be accurately localized using static spatial anchors. Because the object has moved beyond the effective sensing area of the underlying feature extractor, motion regression algorithms have no contextual anchor to calculate where the object went.

This problem is acutely visible in three common alignment strategies:

  1. Deformable Convolutions: These networks learn spatial offsets to dynamically sample features from neighboring frames. However, as Xinyi Zhang et al. and Shi Guo et al. point out, implicit alignment mechanisms like deformable convolutions suffer from a restricted receptive field. If the initial search window does not overlap with the displaced object, the network cannot determine the correct direction to shift its sampling offsets, causing the alignment loop to fail.
  2. Adaptive Convolutions and Kernel-Based Synthesis: Methods that predict spatially varying kernels to blend pixels are strictly bounded by the physical dimensions of the kernel. As Long Mai et al. demonstrate, when an object's displacement exceeds the spatial support window of the predicted kernel, the model cannot capture the underlying movement.
  3. Dilated Convolutions: While dilating kernels expands their field of view without increasing parameter counts, Takashi Isobe notes that convolutional networks still struggle with extremely large or non-linear motion because the sparse sampling pattern of dilated filters can easily skip over fast-moving boundaries entirely.

The Resolution and Temporal Gap Multipliers

The challenge of large motion displacement has been severely amplified by two modern trends: the transition to ultra-high-resolution (4K/8K) content and the processing of wider temporal gaps.

Low Resolution (e.g., 1080p)
[ Object ] ---> 40-pixel shift (Fits within local receptive field)

High Resolution (e.g., 4K)
[     Object     ] --------------> 160-pixel shift (Exceeds local receptive field)

As video resolutions scale up, the absolute pixel distance traveled by an object with the same physical velocity scales linearly. Zhaoyang Jia et al. explain that networks trained on standard low-resolution patches fail on 4K content because the long-range spatial displacements vastly exceed their local receptive fields. This resolution-receptive field mismatch is a primary bottleneck in high-resolution video frame interpolation models, such as those studied by Junheum Park et al. and Hyeonjun Sim et al., where objects routinely shift by hundreds of pixels between frames.

A similar scaling problem occurs when the temporal distance between frames increases. In hierarchical neural video compression, frames are spaced further apart to maximize compression efficiency. Yuxi Liu et al. show that traditional motion estimation networks lack the receptive field required to track objects across these large temporal gaps. Outside of compression, this issue manifests when trying to interpolate between "near-duplicate" photos taken seconds apart. As Fitsum Reda et al. observe, traditional video frame interpolation techniques assume high frame rates where motion is minimal and local; they fail completely when confronted with the massive motion gaps inherent in casual photography.

Architectural Solutions for Global Correspondence

To break free from the constraints of fixed, local receptive fields, recent architectures employ strategies designed to capture global context or dynamically scale their search capacity.

Global and Sparse Transformers

Rather than limiting pixel matching to a local neighborhood, Transformer-based architectures utilize self-attention to look across the entire image. For instance, Zhen Li et al. leverage global pixel correlations to handle large displacements and occlusions effectively.

Because global attention is computationally expensive—especially on 4K frames—researchers use sparse or guided variations. Jing Lin et al. introduce a flow-guided sparse Transformer that uses an initial optical flow estimate to anchor a sparse self-attention mechanism, allowing the network to intelligently sample sharp patches across distant regions without computing an all-pairs pixel matrix.

State Space Models

Selective State Space Models (SSMs), such as Mamba, offer an alternative path to global context with linear computational complexity. Guozhen Zhang et al. adapted these models for video frame interpolation, using interleaved token rearrangement to allow the network to scan across long-range spatial paths, effectively capturing large displacements that would overwhelm standard convolutional networks.

Coarse-to-Fine and Multi-Scale Alignment

Another prevalent strategy relies on a divide-and-conquer approach to scale. Instead of searching for a highly displaced pixel in a single step, models use a multi-stage pipeline.

Shi Guo et al. propose a differentiable two-stage alignment scheme where a patch-level coarse alignment first accounts for large spatial shifts, bringing the target regions into a shared local window where a second, pixel-level refinement stage can safely operate.

Continuous and Generative Representations

Some of the most radical solutions abandon discrete pixel-matching entirely:

  • Geometric Surfaces: In Surface-Aware Feed-Forward Quadratic Gaussian for Frame Interpolation with Large Motion, the authors treat video frames as continuous 3D differential surfaces rather than 2D pixel grids. By mapping frames to Quadratic Gaussians and aligning them using geometric properties like curvature and normals, they bypass the limitations of local pixel neighborhoods.
  • Generative Priors: Rather than trying to mathematically track a pixel across a massive gap, Liao Shen et al. leverage the deep semantic priors of pre-trained diffusion models. By utilizing generative synthesis anchored by semantic understanding, their approach creates smooth transitions between widely separated snapshots, turning a fragile tracking problem into a guided generation problem.

Key Foundational Work

PWC-Net: CNNs for Optical Flow Using Pyramid, Warping, and Cost Volume (2018) — contributed 1 concept elements to this direction.
Object Detection in Videos with Tubelet Proposal Networks (2017) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_large_scale_motion_displacement_96ca7551,
  title = {large-scale motion displacement},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/large-scale-motion-displacement/rdr_96ca75516c784b479ea4ddc212f23daa},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_large_scale_motion_displacement_96ca7551_v1,
  title = {large-scale motion displacement},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/large-scale-motion-displacement/rdr_96ca75516c784b479ea4ddc212f23daa/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake