RAFT: Recurrent All-Pairs Field Transforms for Optical Flow

Zachary Teed, Jia Deng

Introduction

Optical flow estimation—the task of calculating per-pixel motion between consecutive video frames—is a cornerstone of computer vision. Historically, deep learning models for optical flow have relied on "coarse-to-fine" pyramid architectures. While effective, these models often struggle with small, fast-moving objects and error propagation across scales. RAFT (Recurrent All-Pairs Field Transforms) introduces a paradigm shift by maintaining a high-resolution flow field throughout the estimation process, iteratively refined via a recurrent unit.

Architecture and Methodology

The RAFT architecture departs from traditional warping-based methods by using a static 4D correlation volume and a recurrent update operator. It consists of three primary stages:

  1. Feature and Context Extraction: Two separate encoders process the input frames. The feature encoder extracts dense features from both frames, while the context encoder extracts features solely from the first frame to guide the refinement process.
  2. 4D Correlation Volumes: RAFT computes the visual similarity between all pairs of pixels in the two input frames. This results in a 4D correlation volume, which is then pooled at multiple scales to form a correlation pyramid. This "all-pairs" approach ensures that the model can capture both small and large displacements without losing high-resolution detail.
  3. Iterative Recurrent Updates: A Convolutional Gated Recurrent Unit (GRU) iteratively updates the flow field. In each step, it performs a "lookup" in the correlation pyramid based on the current flow estimate. This mimics the steps of a traditional optimization algorithm, allowing the model to converge on a precise flow field over multiple iterations (typically 12–32).
RAFT Architecture: Feature and context encoders feed into a 4D correlation volume, which is iteratively sampled by a recurrent update operator to produce the final optical flow.

Key Innovations

All-Pairs Correlation and Lookups

Unlike previous methods that warp features at each level of a pyramid, RAFT constructs a single, comprehensive correlation volume. By using a lookup operator instead of warping, the optimization landscape remains stable, making it easier for the GRU to learn effective update rules.

The correlation pyramid ($C^1, C^2, C^3$) captures multi-scale similarity information between Image 1 and Image 2, enabling the model to handle various motion magnitudes.

Recurrent Refinement

The use of a GRU with tied weights across iterations makes RAFT highly parameter-efficient. This iterative approach allows the model to "correct" its own errors, a significant advantage over single-pass feed-forward networks.

Convex Upsampling

To recover full resolution from the internal 1/8th resolution, RAFT employs a learned convex upsampling module. This technique predicts a weighted combination of neighboring pixels, leading to sharper motion boundaries compared to standard bilinear interpolation.

Performance and Results

RAFT achieved state-of-the-art results across major benchmarks, often by a wide margin:

  • MPI Sintel: Reduced end-point-error (EPE) on the "Final" pass by 30% compared to previous bests.
  • KITTI-2015: Achieved a 16% reduction in F1-all error.
  • Generalization: RAFT demonstrated exceptional robustness; when trained only on synthetic data (FlyingChairs/FlyingThings3D) and tested on real-world KITTI data, it outperformed previous models by 40%.

Conclusion

RAFT represents a significant leap forward in optical flow estimation. By abandoning the coarse-to-fine bottleneck in favor of high-resolution recurrent refinement and all-pairs correlation, it provides a more robust and accurate framework for motion analysis. Its efficiency and strong generalization capabilities make it a versatile tool for various downstream tasks in video processing and autonomous navigation.

Summary updated

BibTeX
@inproceedings{teed2020raft,
  author = {Zachary Teed and Jia Deng},
  title = {{RAFT: Recurrent All-Pairs Field Transforms for Optical Flow}},
  booktitle = {European Conference on Computer Vision},
  year = {2020},
  doi = {10.1007/978-3-030-58536-5\_24},
}
Figure