Learning Motion Directly from Raw Video Frames

This direction is merged into another direction: motion estimation overhead. Its last version stays readable and citable here.

Version 1 of 126 papers26 conceptsfrozen 2026-10-04

For years, the gold standard for video understanding was the "two-stream" architecture. As popularized by Christoph Feichtenhofer, Andrew Zisserman, and Axel Pinz in Convolutional Two-Stream Network Fusion for Video Action Recognition, these models processed appearance (RGB frames) and motion (pre-computed optical flow) in separate branches.

While effective, this approach has a fatal flaw: optical flow is computationally expensive to calculate and massive to store. This "optical flow bottleneck" prevents models from running in real-time and stops them from being truly end-to-end. Recent research has focused on a singular goal: teaching neural networks to "see" motion directly from raw pixels, bypassing the need for external motion estimation entirely.

The Problem with Explicit Motion

Optical flow represents the displacement of every pixel between two frames. Calculating it usually requires solving a complex optimization problem for every pair of frames in a video. As noted by Ali Diba and colleagues in Deep Temporal Linear Encoding Networks, this dependency makes scaling to large datasets or real-time applications nearly impossible.

To solve this, researchers have moved toward "in-network" motion representation—architectures that extract temporal dynamics as naturally as they extract spatial textures.

Factorizing Space and Time

The most direct way to capture motion is through 3D convolutions ($XYT$), which slide a cube-shaped filter over a stack of frames. However, 3D kernels are parameter-heavy and difficult to train.

A key breakthrough came from Kui Jia and his team in Human Action Recognition Using Factorized Spatio-Temporal Convolutional Networks. They proposed factorizing 3D convolutions into sequential 2D spatial filters followed by 1D temporal filters. This $2D+1D$ approach reduces complexity while allowing the model to learn how features change over time. By breaking the problem down, the network can use pre-trained 2D image weights for the spatial part and focus its learning capacity on the temporal "residual."

Simulating Flow with Feature Shifting

If optical flow is just the difference between frames, can we approximate it inside the network? Myunggi Lee and Nojun Kwak explored this in Motion Feature Network: Fixed Motion Filter for Action Recognition.

Their "Motion Block" uses a clever trick: it shifts feature maps from consecutive frames in different directions and subtracts them from the original map. This operation mimics the basic principle of optical flow—detecting where a feature has moved—but does so using simple, differentiable operations that are much faster than traditional flow algorithms. Similarly, Shichao Zhao et al. utilized a "frame-diff" layer in Pooling the Convolutional Layers in Deep ConvNets for Action Recognition to capture these immediate temporal changes.

Temporal Excitation and Attention

Another intuition is that not all parts of a video are equally important for understanding an action. In TEA: Temporal Excitation and Aggregation for Action Recognition, Yan Li and Jianguo Zhang introduced a "Temporal Excitation" (TE) module.

The TE module calculates the difference between adjacent frames to identify "moving" regions and then uses this information to "excite" (amplify) the corresponding features in the current frame. This acts as a form of temporal attention, telling the network: "Pay more attention to the pixels that are changing." This allows standard 2D networks to become motion-aware with minimal extra computation.

Learning to Predict the Future

Perhaps the most sophisticated way to understand motion is to predict it. Xia Huang and Gemma Roig applied "Predictive Coding" to this problem in Predictive Coding Networks Meet Action Recognition.

Instead of just classifying a frame, their model tries to predict the features of the next frame. To do this successfully, the network must develop an internal model of how objects move. If the network can predict where a hand will be in the next millisecond, it has inherently mastered the representation of motion.

Beyond Action Recognition

This shift away from explicit motion estimation isn't limited to action recognition. In video denoising, Julie Delon and Matias Tassano developed FastDVDnet, which uses a multi-scale CNN to align frames implicitly. By avoiding the errors and costs of explicit flow, they achieved real-time speeds and cleaner results. Similarly, Xing Zhao used motion-aware memory networks for salient object detection in Motion-aware Memory Network for Fast Video Salient Object Detection, showing that temporal context can be stored and retrieved without pixel-perfect alignment.

Conclusion

The trajectory of the field is clear: we are moving from "hand-crafted" motion features like optical flow toward "learned" motion features. By integrating temporal reasoning directly into the architecture—whether through factorization, feature shifting, or predictive coding—researchers like Ahsan Iqbal and Suman Saha are building systems that are not only faster but more robust. The future of video understanding lies in networks that don't need to be told how things move, but rather, learn to see the motion for themselves.

Papers in this version

History & citationVersion 1 · 2026-10-04

Merged into another direction · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_learning_motion_directly_from_raw_video__a617c514,
  title = {Learning Motion Directly from Raw Video Frames},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/learning-motion-directly-from-raw-video-frames/rdr_a617c514307f4e7e857d6f895c522671},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_learning_motion_directly_from_raw_video__a617c514_v1,
  title = {Learning Motion Directly from Raw Video Frames},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/learning-motion-directly-from-raw-video-frames/rdr_a617c514307f4e7e857d6f895c522671/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}