RAFT: Recurrent All-Pairs Field Transforms for Optical Flow (Extended Abstract)

Zachary Teed, Jia Deng

Introduction: Beyond Coarse-to-Fine Pyramids

Optical flow—the task of estimating per-pixel motion between two video frames—is a fundamental challenge in computer vision. For years, the dominant deep learning paradigm followed a "coarse-to-fine" strategy: estimating flow at low resolutions and upsampling it through a pyramid of layers.

However, this approach has a major flaw: if the model makes an error at a coarse resolution (such as missing a small, fast-moving object), that error propagates and is nearly impossible to fix at finer scales. Additionally, these models often struggle to generalize to new types of scenes because they are over-optimized for specific dataset characteristics. RAFT (Recurrent All-Pairs Field Transforms) abandons this traditional approach in favor of a recurrent architecture that maintains high-resolution features throughout the estimation process.

The RAFT Architecture

RAFT introduces a fundamentally different architecture composed of three main stages designed to refine flow iteratively at a single, high resolution.

  1. Feature & Context Extraction: The model uses two convolutional encoders. One extracts feature vectors for every pixel in both input images. The second (the context encoder) extracts features only from the first image to provide spatial context for the refinement process.
  2. All-Pairs Correlation Volume: Instead of searching for matches in a limited neighborhood, RAFT computes the visual similarity between every pixel in the first image and every pixel in the second. This results in a 4D "correlation volume." This volume is then pooled into a 4-layer pyramid to provide information about both large and small displacements while keeping the first image's resolution high.
  3. Recurrent Update Operator: This is the "brain" of RAFT. It uses a gated recurrent unit (GRU) to iteratively update the flow field. In each iteration, it performs a "lookup" in the correlation volume based on the current flow estimate and uses that information to propose a refinement. This mimics the behavior of traditional optimization algorithms but is entirely learned from data.
The RAFT architecture consists of feature/context encoders, an all-pairs correlation volume, and a recurrent GRU-based update operator.

Design Principles and Innovations

  • High Resolution is King: By operating at a single high resolution throughout the entire process, RAFT avoids the "error propagation" trap of coarse-to-fine pyramids. It can recover the motion of small, fast-moving objects that disappear at lower resolutions.
  • Optimization as a Recurrent Process: The GRU-based update operator effectively learns to perform an optimization. Because the weights are "tied" (the same weights are used for every iteration), the model is lightweight and encourages convergence to a stable flow field.
  • The Power of All-Pairs: Computing all-pairs correlations allows the model to find matches across the entire image, making it much more robust to large displacements compared to models with a restricted search window.
  • Strong Generalization: One of the most striking findings was RAFT's "zero-shot" performance. When trained only on synthetic data, it outperformed previous models that had been specifically fine-tuned on the target datasets.

Experimental Performance

The authors trained and evaluated RAFT using several standard benchmarks, including synthetic pre-training on FlyingChairs and FlyingThings3D, and real-world evaluation on Sintel and KITTI-2015.

  • State-of-the-Art Accuracy: On the KITTI benchmark, RAFT reduced the error (F1-all) by 16% compared to the previous best result. On Sintel (Final pass), it achieved a 30% error reduction.
  • Efficiency: Despite its complexity, RAFT is highly efficient. It processes 1088 × 436 videos at 10 frames per second on a standard 1080Ti GPU. A smaller, optimized version runs at 20 fps.
  • Training Speed: RAFT requires significantly fewer iterations to converge—often 10X fewer than competing architectures like PWC-Net or FlowNet2.
Qualitative results of RAFT on various challenging video sequences, demonstrating precise flow estimation for complex motions.

Limitations and Future Work

While RAFT is a massive leap forward, some ongoing challenges remain. Because the model is recurrent, the inference time scales with the number of iterations. While 32 iterations are standard for high accuracy, real-time applications might need to sacrifice some precision for speed. Furthermore, while RAFT handles occlusions better than previous models, extreme motion blur and long-term occlusions remain difficult for any per-pixel matching strategy.

The paper suggests that the "learning to optimize" framework used in RAFT could be applied to other dense correspondence problems in vision, such as stereo matching or 3D reconstruction, potentially revolutionizing how we approach spatial reasoning in video.

Summary updated

Abstract

We introduce Recurrent All-Pairs Field Transforms (RAFT), a new deep network architecture for optical flow. RAFT extracts per-pixel features, builds multi-scale 4D correlation volumes for all pairs of pixels, and iteratively updates a flow field through a recurrent unit that performs lookups on the correlation volumes. RAFT achieves state-of-the-art performance on the KITTI and Sintel datasets. In addition, RAFT has strong cross-dataset generalization as well as high efficiency in inference time, training speed, and parameter count.
BibTeX
@inproceedings{teed2021raft,
  author = {Zachary Teed and Jia Deng},
  title = {{RAFT: Recurrent All-Pairs Field Transforms for Optical Flow (Extended Abstract)}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2021},
  doi = {10.24963/ijcai.2021/662},
}
Figure