Abstract
Despite the significant progress that has been made on estimating optical flow recently, most estimation methods, including classical and deep learning approaches, still have difficulty with multi-scale estimation, real-time computation, and/or occlusion reasoning. In this paper, we introduce dilated convolution and occlusion reasoning into unsupervised optical flow estimation to address these issues. The dilated convolution allows our network to avoid upsampling via deconvolution and the resulting gridding artifacts. Dilated convolution also results in a smaller memory footprint which speeds up inference. The occlusion reasoning prevents our network from learning incorrect deformations due to occluded image regions during training. Our proposed method outperforms state-of-the-art unsupervised approaches on the KITTI benchmark. We also demonstrate its generalization capability by applying it to action recognition in video.
Introduction
Optical flow—the task of estimating per-pixel motion between video frames—is a cornerstone of computer vision. While Deep Learning has pushed the field forward, three major hurdles remain:
- The Data Gap: Supervised models require "ground truth" flow, which is nearly impossible to capture for real-world scenes, forcing a reliance on synthetic data that doesn't always generalize well.
- Resolution Loss & Artifacts: Standard CNNs downsample images to extract features and then "upsample" them back using deconvolution. This often leads to "checkerboard artifacts" and the loss of fine-grained structural details.
- The Occlusion Problem: Unsupervised methods usually assume that every pixel in Frame A must appear in Frame B (brightness constancy). When an object moves behind another (occlusion), this assumption fails, providing the network with "garbage" gradients that degrade training.
Methodology
The authors propose a bidirectional unsupervised learning framework with three key architectural innovations:
Dilated Convolutions
Instead of using pooling and deconvolution, the network uses dilated (or "atrous") convolutions in the later stages. This allows the network to expand its receptive field (seeing more of the image context) without reducing the spatial resolution of the feature maps. This eliminates the need for upsampling and prevents gridding artifacts.
Explicit Occlusion Reasoning
The model estimates both forward and backward flow simultaneously. It performs a "consistency check": if the forward and backward vectors for a pixel don't match up (within a certain threshold), that pixel is flagged as occluded. Crucially, the loss function ignores these occluded pixels, preventing the network from learning incorrect deformations.
Dense Connectivity
Inspired by DenseNet, the authors use dense connections between layers to help the network capture thin structures and very small displacements that are often lost in sparser architectures.
Experimental Setup
The model was trained and evaluated using:
- SYNTHIA: A large synthetic dataset of driving scenes used for initial pre-training.
- KITTI 2012 & 2015: Real-world driving benchmarks used for fine-tuning and final evaluation.
- UCF101: A classic action recognition dataset used to demonstrate the model's ability to generalize to tasks beyond simple flow estimation.
Results and Discussion
Key Insights
- Resolution is King: By avoiding the "bottleneck" of heavy downsampling, the network retains the high-frequency details necessary for accurate flow, especially around object boundaries.
- Less is More (Parameters): Because the dilated network avoids a complex deconvolutional decoder, it has roughly half the parameters of the standard FlowNetS architecture. This makes it significantly faster and less memory-intensive.
- Cleaning the Gradients: By masking out occlusions, the training signal becomes much "cleaner," allowing the model to focus on learning valid motion.
Key Results
The method outperformed all previous unsupervised approaches on the KITTI benchmarks. On KITTI 2012, it achieved a non-occluded pixel error (AEE NOC) of 2.35, a significant lead over earlier unsupervised models. Furthermore, the model performs inference at 33.3 frames per second (fps), which is roughly 4x faster than FlowNet2. When applied to action recognition on UCF101, the flow generated by this model resulted in 82.5% accuracy, outperforming the supervised FlowNet2 (79.51%).
Conclusion
While the model is highly efficient, its performance still lags slightly behind the very best supervised models. Future work could involve "stacking" multiple versions of the network to bridge this gap. Additionally, improving robustness to extreme lighting changes beyond the current census transform remains an area for exploration.
Summary updated