Efficient Motion Representation Without Explicit Optical Flow

This direction is merged into another direction: motion estimation overhead. Its last version stays readable and citable here.

Version 1 of 1239 papers247 conceptsfrozen 2026-10-04

For years, the "gold standard" for teaching computers to understand movement in video has been optical flow. By calculating a vector for every single pixel to describe where it moves in the next frame, researchers could provide models with a rich, explicit map of temporal dynamics. However, this precision comes with a heavy "tax": calculating dense optical flow is often the single most expensive part of a video processing pipeline, frequently taking 50 to 100 times longer than the actual neural network inference.

A growing body of research seeks to break this dependency. The goal is to capture the benefits of motion—temporal consistency and action cues—without the crippling computational overhead of traditional flow estimation.

The Optical Flow Bottleneck

The fundamental problem with optical flow is that it treats motion as a dense, per-pixel optimization task. As noted by Wenguan Wang and Jianbing Shen, this creates a massive bottleneck that prevents high-accuracy models from reaching real-time speeds in tasks like salient object detection. Even in modern pipelines, Yixiao Li and Weide Liu point out that pre-processing time for flow can exceed inference time by a factor of fifty, making it impractical for high-throughput video quality assessment.

To solve this, researchers have converged on three primary strategies: leveraging "free" motion data from video codecs, estimating motion at the feature level rather than the pixel level, and replacing explicit vectors with lightweight attention mechanisms.

Leveraging the Compressed Domain

The most direct way to avoid calculating motion is to use motion that has already been calculated. Most videos are stored in compressed formats (like H.264 or HEVC) that already contain Motion Vectors (MVs) used for inter-frame compression.

Chao-Yuan Wu pioneered this with CoViAR, training networks directly on these "free" signals. While motion vectors are coarser and noisier than optical flow, they provide a sufficient prior for action recognition while running nearly 5x faster. Similarly, the Inter-BMV framework uses these block motion vectors to interpolate features between keyframes, achieving significant speedups in semantic segmentation by avoiding redundant per-frame inference. More recently, Mert Onur Cakiroglu and Hasan Kurban demonstrated that these embedded vectors can even be used to evaluate the "temporal realism" of AI-generated videos, providing a lightweight alternative to heavy flow-based benchmarks.

Feature Flow and Sparse Propagation

If we cannot use the codec's motion, the next most efficient approach is to move the motion estimation inside the neural network and perform it on downsampled data.

Xizhou Zhu and Jifeng Dai introduced Deep Feature Flow (DFF), which runs expensive feature extraction only on sparse "key frames." They then use a lightweight flow network to "warp" these high-level features to intermediate frames. Because feature maps are spatially smaller than raw images, this warping is incredibly fast.

Taking this further, Guosheng Lin and Ruibing Jin developed IFF-Net, which estimates "in-network" feature flow. By ditching external, high-capacity flow estimators in favor of a module that learns motion directly in the feature space, they achieved state-of-the-art detection accuracy with a much smaller memory footprint.

Motion as Attention and Matching

A third direction replaces the concept of "vectors" entirely, treating motion as a correspondence problem that can be solved via attention or matching.

  • Sparse Attention: Chaoxu Guo and Bin Fan proposed Progressive Sparse Local Attention (PSLA). Instead of calculating where a pixel goes, the model learns to attend to relevant regions in previous frames using progressively sparser strides. This biologically inspired mechanism captures temporal dependencies without the need for an explicit flow field.
  • Global Matching: In video object segmentation, Tao Zhuo and Mohan Kankanhalli replaced flow with a global pixel matching strategy. By matching features across frames on downsampled maps, their Mask Transfer Network (MTN) runs at 37 FPS—hundreds of times faster than methods relying on fine-tuning or dense flow.
  • Structural Similarity: Kalana Abeywardena and Ranga Rodrigo showed that even simple Structural Similarity (SSIM) maps can serve as a proxy for motion in action localization, allowing their KORSAL framework to maintain high frame rates on standard hardware.

Learning Motion Internally

Finally, some researchers argue that if a model is deep enough, it can learn to represent motion implicitly. Ali Diba and Luc Van Gool developed 3D CNNs with integrated deconvolutional layers that learn to estimate motion internally. By training the network to understand appearance and motion simultaneously, they eliminated the need for any pre-computed flow, achieving a 20x speedup over traditional two-stream methods.

This shift—from viewing motion as a pre-processing requirement to viewing it as an internal, learnable representation—is what allows modern video AI to move from the lab into real-time applications like autonomous navigation and live surveillance. As Jinyoung Moon and Sunah Min demonstrate with their Information Elevation Network, the future of video analysis lies in "lifting" relevant temporal information directly from RGB frames, leaving the heavy burden of optical flow behind.

Papers in this version

Showing 60 of 239 papers.

History & citationVersion 1 · 2026-10-04

Merged into another direction · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_efficient_motion_representation_without__ad3de05c,
  title = {Efficient Motion Representation Without Explicit Optical Flow},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/efficient-motion-representation-without-explicit-optical-flow/rdr_ad3de05c37ef4d0da5218bee21fcd597},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_efficient_motion_representation_without__ad3de05c_v1,
  title = {Efficient Motion Representation Without Explicit Optical Flow},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/efficient-motion-representation-without-explicit-optical-flow/rdr_ad3de05c37ef4d0da5218bee21fcd597/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}