AccFlow: Backward Accumulation for Long-Range Optical Flow

Guangyang Wu, Xiaohong Liu 0001, Kunming Luo, Xi Liu, Qingqing Zheng, Shuaicheng Liu, Xinyang Jiang, Guangtao Zhai, Wenyi Wang

Abstract

Estimating optical flow between distant video frames remains a significant challenge due to increasing occlusions and compounding accumulation errors. We present AccFlow, a novel framework that utilizes a "backward accumulation" strategy to estimate long-range motion. By recursively fusing local flows in reverse order and employing a deformable AccPlus module alongside an adaptive blending mechanism, the framework effectively manages compounding errors and increasing occlusions in distant frames. Our approach demonstrates substantial improvements on the CVO and HS-Sintel benchmarks, particularly in complex, fast-moving scenes.

Introduction

Optical flow—the task of tracking pixel motion between frames—is well-solved for consecutive frames (local flow). However, estimating long-range optical flow (e.g., between frame 1 and frame 10) remains a major challenge. As the temporal gap between frames increases, two primary issues emerge:

  1. Occlusions increase: Objects move behind one another, making it impossible to find direct correspondences.
  2. Accumulation errors compound: Traditional "forward accumulation" (chaining local flows from the reference frame) causes small errors in each step to snowball. Furthermore, the area considered "occluded" relative to the first frame grows progressively larger, eventually overwhelming the estimator.

Methods

The AccFlow framework is built on three primary technical pillars designed to address the limitations of long-range flow estimation.

Backward Accumulation

Unlike traditional methods that accumulate flow from the reference frame forward ($1 \to 2, 1 \to 3, \dots$), AccFlow reverses this process. It starts from the target frame and works backward. The mathematical insight is that by aligning the large accumulated flow to the smaller local flow at each step, the "occlusion proportion" remains stable and minimal, rather than growing uncontrollably.

AccPlus and Adaptive Blending

The AccPlus module serves as the core engine for motion accumulation, utilizing deformable convolutions in the feature domain to align motion features accurately. It includes an internal "occlusion solver" that leverages spatial context to fill in gaps. To prevent recursive drift, the Adaptive Blending module incorporates a "direct" (but potentially noisy) estimate of the long-range flow as a prior, using an attention-based confidence mask to blend the accumulated flow with this direct estimate.

Key Insights: The Geometry of Accumulation

The most significant contribution of this work is the demonstration that backward accumulation is superior to forward accumulation. In forward accumulation, the occlusion mask grows monotonically with time. In contrast, with backward accumulation, the reference frame effectively "resets" at each step of the recursion. This keeps the occlusion area small and manageable, making the task significantly easier for the neural network to learn and execute.

Comparison of occlusion masks in forward vs. backward accumulation.

Experimental Evaluation

The authors introduced two new datasets to evaluate long-range flow: CVO (Cross-frame Video Optical flow), a large-scale synthetic dataset, and HS-Sintel, a high-speed version of the classic Sintel dataset.

AccFlow was evaluated by integrating standard estimators like RAFT and GMA as its backbone. On the CVO (Final) benchmark, the Acc+GMA configuration achieved an Average End-Point Error (EPE) of 2.808, a massive improvement over the 6.265 EPE of the vanilla GMA model. The method particularly excels in occluded regions, reducing EPE in the HS-Sintel dataset by nearly 30% compared to previous state-of-the-art methods. Qualitative results confirm that AccFlow maintains sharp object boundaries and successfully recovers small objects with large motions that other methods often lose.

Qualitative comparison of AccFlow against RAFT and GMA on the Sintel dataset.

Conclusion and Future Work

AccFlow represents a significant step forward in long-range motion estimation. While highly effective, the recursive nature of the framework introduces computational overhead; for a 7-frame sequence, inference takes between 0.8s and 1.5s on an NVIDIA GTX 3090. Future work will focus on developing more efficient architectures to reduce inference time and testing the framework's limits on even longer sequences spanning dozens or hundreds of frames.

Summary updated

Abstract

Recent deep learning-based optical flow estimators have exhibited impressive performance in generating local flows between consecutive frames. However, the estimation of long-range flows between distant frames, particularly under complex object deformation and large motion occlusion, remains a challenging task. One promising solution is to accumulate local flows explicitly or implicitly to obtain the desired long-range flow. Nevertheless, the accumulation errors and flow misalignment can hinder the effectiveness of this approach. This paper proposes a novel recurrent framework called AccFlow, which recursively backward accumulates local flows using a deformable module called as AccPlus. In addition, an adaptive blending module is designed along with AccPlus to alleviate the occlusion effect by backward accumulation and rectify the accumulation error. Notably, we demonstrate the superiority of backward accumulation over conventional forward accumulation, which to the best of our knowledge has not been explicitly established before. To train and evaluate the proposed AccFlow, we have constructed a large-scale high-quality dataset named CVO, which provides ground-truth optical flow labels between adjacent and distant frames. Extensive experiments validate the effectiveness of AccFlow in handling long-range optical flow estimation. Codes are available at https://github.com/mulns/AccFlow .
BibTeX
@inproceedings{wu2023accflow,
  author = {Guangyang Wu and Xiaohong Liu and Kunming Luo and Xi Liu and Qingqing Zheng and Shuaicheng Liu and Xinyang Jiang and Guangtao Zhai and Wenyi Wang},
  title = {{AccFlow: Backward Accumulation for Long-Range Optical Flow}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2023},
}
Figure