Abstract
This paper presents Multi-Scale Correspondence Structure Learning (MSCSL), an end-to-end deep learning framework designed for robust optical flow estimation. By treating image scales as a sequence and employing a Spatial Conv-GRU, the model adaptively captures dependencies across resolutions. This approach effectively addresses the challenges of scale variance and error propagation inherent in traditional coarse-to-fine strategies. Evaluated on major benchmarks like MPI Sintel and KITTI 2012, MSCSL demonstrates superior accuracy and competitive real-time performance.
Introduction
Optical flow estimation—the process of determining pixel-wise motion between consecutive video frames—is fundamental to numerous computer vision tasks. However, it remains challenging due to two primary factors: scale variance and the limitations of hierarchical processing. Real-world motion encompasses a wide range of displacements, from large jumps to subtle shifts. Traditional "coarse-to-fine" methods often fail because errors introduced at lower resolutions are magnified during subsequent refinement stages.
To overcome these "pyramid pitfalls," the authors propose MSCSL. The core idea is to model "cross-scale dependency," allowing the network to reason across multiple resolutions simultaneously. This ensures that global motion context and local fine-grained details remain consistent throughout the estimation process.
Methodology
The MSCSL framework is structured into three distinct stages:
- Multi-Scale Feature Extraction: A Siamese CNN architecture extracts features from the input image pair at various levels. Coarse features provide high-level semantic information for large displacements, while fine features preserve local textures necessary for precise small-scale motion.
- Correspondence Structure Modeling: The network computes correlation maps at each scale, measuring the similarity between patches in the first image and potential matches in the second.
- Spatial Conv-GRU Encoding: This is the central innovation of the paper. Instead of modeling temporal sequences, a Gated Recurrent Unit (GRU) is used to model the sequence of scales. This Spatial Conv-GRU processes correlation maps from coarse to fine, utilizing internal convolutions and upsampling to maintain spatial topology. This allows information from broader scales to intelligently guide the refinement at higher resolutions.
The model is optimized using a hybrid loss function combining supervised End-Point Error (EPE) and an unsupervised reconstruction loss based on brightness constancy. Notably, the unsupervised loss is applied in the feature space rather than the pixel space, enhancing robustness against noise and illumination changes.
Experiments and Results
The MSCSL model was evaluated on several prominent benchmarks, including Flying Chairs, MPI Sintel, KITTI 2012, and Middlebury.
Key Insights
- Scale as a Sequence: Replacing rigid pyramids with an RNN-based sequential approach allows the network to maintain global context while focusing on local details.
- Feature-Space Reconstruction: Applying unsupervised constraints to intermediate feature maps ($Y^A, Y^B$) significantly improves resilience to environmental factors like shadows and reflections.
- Adaptive Fusion: The Conv-GRU adaptively integrates information across scales, leading to smoother and more accurate flow fields compared to simple feature concatenation.
Performance Comparison
MSCSL outperformed established models such as FlowNet and SPyNet across multiple metrics. On the MPI Sintel (Clean) test set, MSCSL achieved an EPE of 5.78, a notable improvement over FlowNetS (6.97). On the real-world KITTI 2012 dataset, it reduced the EPE to 5.77 from FlowNetS's 7.52.
Despite the recurrent architecture, the model remains efficient, processing frames at approximately 0.06 seconds on a Titan X GPU, making it suitable for near-real-time applications.
Conclusion
MSCSL introduces a novel way to handle the multi-scale nature of optical flow by leveraging recurrent neural networks to model scale dependencies. While the model shows state-of-the-art performance on several benchmarks, it still faces challenges in regions with uniform textures where correlation features are sparse. Future work will focus on improving robustness in these textureless areas and handling extremely high-velocity motions more effectively.
Summary updated