Detail Preserving Residual Feature Pyramid Modules for Optical Flow

Libo Long, Jochen LangWACV 2022

TL;DR

This paper introduces a novel Residual Feature Pyramid Module (RFPM) to combat the blurring of object boundaries in optical flow estimation caused by traditional feature pyramid downsampling. By incorporating a residual structure with max pooling and learnable "repair masks," RFPM preserves fine details, leading to significantly improved accuracy on challenging datasets like Sintel and KITTI, while also enabling efficient fine-tuning.

What Problem Does This Paper Address?

Estimating optical flow – the motion of objects between consecutive frames in a video – is crucial for many computer vision tasks. Modern methods often rely on iterative refinement using feature pyramids, where images are processed at multiple resolutions. However, a significant challenge arises during the downsampling process in these pyramids: it can blur the boundaries of fine details and small objects, causing them to blend with the background. This "boundary blur" misleads the flow estimation, leading to errors that can be amplified in subsequent refinement steps. The paper aims to address this fundamental limitation by preserving these critical boundary details within the feature pyramids.

Methods

The core contribution is the Residual Feature Pyramid Module (RFPM). RFPM enhances traditional feature pyramids in two key ways:

  • Residual Feature Downsampling (RFD): Instead of solely relying on standard weighted feature downsampling (WFD), RFD combines WFD with Max Pooling (MP). The idea is that MP can help retain higher-value pixels, preserving more detail. RFD uses a residual connection to guide the WFD module with information from MP, aiming to enhance edge areas.
  • Repair Masks (RM): RFPM introduces learnable "repair masks" that are applied at specific levels of the feature pyramid. These masks have two components: an attention mask and a bias mask. They are designed to "repair" the information lost during downsampling, particularly due to max pooling, and to enhance the clarity of features at object boundaries.

RFPM is designed to be modular, allowing it to be integrated into existing iterative refinement optical flow architectures without major modifications. The paper demonstrates its integration with two state-of-the-art methods: IRR-PWC and RAFT. They also propose an efficient fine-tuning strategy that significantly reduces training time by incorporating RFPM into pre-trained models with a smaller training schedule and novel data augmentation.

Datasets

The paper uses the following datasets:

  • FlyingChairs_OCC: Used for initial training.
  • FlyingThings3D: Used for training and fine-tuning.
  • MPI Sintel: Used for evaluation (Clean and Final passes).
  • KITTI-2015: Used for evaluation (two-frame benchmark).
  • HD1K: Used for fine-tuning.

No new benchmarks were introduced.

Key Insights and Ideas

  • Boundary Blur is a Root Cause: The paper convincingly argues that the blurring of object boundaries during feature pyramid downsampling is a primary source of error in optical flow, especially for small and detailed structures.
  • Max Pooling as a Detail Preserver: The insight that max pooling, despite its simplicity, can be a valuable tool for preserving local detail in feature maps when combined with residual learning is key.
  • "Repair Masks" for Information Recovery: The concept of learnable repair masks is novel and effective in actively correcting for information loss during downsampling, leading to sharper feature representations.
  • Modularity for Easy Integration: The RFPM's design as a plug-and-play module makes it highly practical, allowing researchers to easily improve existing optical flow methods.
  • Efficient Fine-Tuning is Crucial: The proposed fine-tuning strategy demonstrates that significant performance gains can be achieved with substantially less training data and time, making advanced methods more accessible.

Key Results

  • State-of-the-Art Performance: RFPM-RAFT achieves state-of-the-art results on the MPI Sintel (Clean pass) and KITTI 2015 benchmarks.
  • Significant Accuracy Improvement: RFPM-RAFT shows a 12% higher accuracy on Sintel and 6% higher accuracy on KITTI compared to the baseline RAFT.
  • Visual Improvements: Qualitative results (Figure 6) clearly show that RFPM-RAFT produces sharper feature maps and more accurate flow predictions around object boundaries compared to RAFT.
  • Efficient Fine-Tuning: The RFPMt-RAFT model, trained with a significantly reduced schedule (22.2% of RAFT's iterations), still surpasses RAFT on KITTI-2015.

Limitations and Future Work

  • Repair Mask Placement: The paper notes that adding repair masks at all levels of the pyramid can lead to overfitting. They found that applying them at lower levels (e.g., levels 1 and 2) is sufficient and optimal.
  • Ablation Study Focus: The ablation study primarily focuses on RFPM-RAFT, suggesting that further exploration of RFPM with other architectures might be beneficial.
  • Future Work: The authors suggest exploring the use of RFPM in other computer vision tasks that rely on feature pyramids and further investigating the benefits of their asymmetric data augmentation strategy.

Summary updated

BibTeX
@inproceedings{long2022detail,
  author = {Libo Long and Jochen Lang},
  title = {{Detail Preserving Residual Feature Pyramid Modules for Optical Flow}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2022},
}
Figure