GAFlow: Incorporating Gaussian Attention into Optical Flow

Ao Luo, Fan Yang, Xin Li, Lang Nie, Chunyu Lin, Haoqiang Fan, Shuaicheng Liu

Introduction

Optical flow estimation—the task of calculating pixel-wise motion between consecutive video frames—requires a delicate balance between two competing objectives: discrimination and smoothness. A model must be discriminative enough to distinguish a pixel from its neighbors for accurate matching, yet smooth enough to ensure that neighboring pixels belonging to the same object move consistently.

While state-of-the-art architectures like RAFT and FlowFormer have advanced the field using Transformers and 4D correlation volumes, they often lack a strong local inductive bias. Global attention mechanisms can introduce noise from unrelated image regions, while rigid local windows fail to capture complex deformations. GAFlow addresses this "neighborhood priority" gap by incorporating learnable Gaussian Attention into both feature encoding and motion refinement.

Proposed Architecture

The Gaussian Attention Flow (GAFlow) network introduces two primary modules designed to be "plug-and-play" within existing flow architectures.

The GAFlow architecture integrates Gaussian constraints into both the feature extraction and motion estimation phases.

1. Gaussian-Constrained Layer (GCL)

Integrated into the feature encoder, the GCL modifies standard Transformer blocks by applying a learnable Gaussian kernel as a constraint mask on the attention matrix. This forces the model to prioritize fine-grained structural details in a pixel's immediate neighborhood, resulting in more distinct features that are easier to match across frames.

2. Gaussian-Guided Attention Module (GGAM)

Used during the motion estimation (decoding) phase, the GGAM provides a learnable smoothing mechanism. Its most advanced iteration, the Gaussian-Guided Attention with Deformation (GGAD), learns dynamic offsets to warp the Gaussian kernel and amplitudes to scale it. This allows the "attention bubble" to adapt its shape and size to match deforming objects or large displacements, ensuring smoothing does not cross object boundaries.

Local Inductive Bias and Feature Peakiness

A key insight of GAFlow is that re-introducing locality as a prior prevents the model from being distracted by global similarities that are irrelevant to motion. This results in significantly more "peaky" feature representations.

Comparison of affinity maps: Gaussian Affinity (left) produces sharp, localized peaks compared to the broad, noisy distribution of standard Global Affinity (right).

As shown in the visualization above, standard global attention models often produce blurry affinity maps with multiple potential matches. In contrast, GAFlow’s Gaussian-constrained approach produces sharp, localized peaks, leading to more accurate and robust pixel matching.

Experimental Results

GAFlow was evaluated on the Sintel and KITTI-2015 benchmarks, setting new performance records while maintaining high computational efficiency.

  • Sintel Benchmark: GAFlow-FF (based on FlowFormer) achieved an End-Point Error (EPE) of 1.15 on Clean and 2.05 on Final, outperforming previous state-of-the-art models.
  • KITTI-2015: The model achieved a top-tier 4.42% F1-all score on the online leaderboard.
  • Efficiency: GAFlow is significantly faster and lighter than its predecessors. On KITTI frames, GAFlow inference takes 0.36s (vs. FlowFormer’s 1.77s) and uses approximately 40% fewer parameters (11.1M vs. 18.2M).

Conclusion

GAFlow demonstrates that incorporating learnable local priors via Gaussian Attention can significantly enhance the performance and efficiency of optical flow models. By making feature matching more discriminative and motion smoothing more adaptive, GAFlow provides a robust framework for dense correspondence tasks. Future work may extend this approach to related domains such as stereo matching and depth estimation.

Summary updated

Abstract

Optical flow, or the estimation of motion fields from image sequences, is one of the fundamental problems in computer vision. Unlike most pixel-wise tasks that aim at achieving consistent representations of the same category, optical flow raises extra demands for obtaining local discrimination and smoothness, which yet is not fully explored by existing approaches. In this paper, we push Gaussian Attention (GA) into the optical flow models to accentuate local properties during representation learning and enforce the motion affinity during matching. Specifically, we introduce a novel Gaussian-Constrained Layer (GCL) which can be easily plugged into existing Transformer blocks to highlight the local neighborhood that contains fine-grained structural information. Moreover, for reliable motion analysis, we provide a new Gaussian-Guided Attention Module (GGAM) which not only inherits properties from Gaussian distribution to instinctively revolve around the neighbor fields of each point but also is empowered to put the emphasis on contextually related regions during matching. Our fully-equipped model, namely Gaussian Attention Flow network (GAFlow), naturally incorporates a series of novel Gaussian-based modules into the conventional optical flow framework for reliable motion analysis. Extensive experiments on standard optical flow datasets consistently demonstrate the exceptional performance of the proposed approach in terms of both generalization ability evaluation and online benchmark testing. Code is available at https://github.com/LA30/GAFlow.
BibTeX
@inproceedings{luo2023gaflow,
  author = {Ao Luo and Fan Yang and Xin Li and Lang Nie and Chunyu Lin and Haoqiang Fan and Shuaicheng Liu},
  title = {{GAFlow: Incorporating Gaussian Attention into Optical Flow}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2023},
}
Figure