Abstract
Classical approaches for estimating optical flow have achieved rapid progress in the last decade. However, most of them are too slow to be applied in real-time video analysis. Due to the great success of deep learning, recent work has focused on using CNNs to solve such dense prediction problems. In this paper, we investigate a new deep architecture, Densely Connected Convolutional Networks (DenseNet), to learn optical flow. This specific architecture is ideal for the problem at hand as it provides shortcut connections throughout the network, which leads to implicit deep supervision. We extend current DenseNet to a fully convolutional network to learn motion estimation in an unsupervised manner. Evaluation results on three standard benchmarks demonstrate that DenseNet is a better fit than other widely adopted CNN architectures for optical flow estimation.
Introduction
Optical flow—the task of estimating per-pixel motion between two consecutive video frames—is a cornerstone of computer vision, essential for applications ranging from autonomous driving to video editing. While classical methods often struggle with real-time performance, deep learning approaches like FlowNet have shifted the focus toward speed and accuracy.
However, most high-performing CNNs rely on supervised learning, requiring massive synthetic datasets with "ground truth" motion labels that often fail to generalize to real-world scenarios. Furthermore, standard architectures tend to lose fine image details as data passes through deep layers. This research addresses two primary challenges: preserving high-frequency details for better motion boundaries and training effectively without human-labeled ground truth.
Architecture and Methodology
The authors propose a Fully Convolutional DenseNet that utilizes a "contracting-expanding" (encoder-decoder) structure.
The core components include:
- Dense Blocks: Instead of traditional convolutional layers, the network employs Dense Blocks where each layer is connected to every subsequent layer. This ensures "maximum information flow" and efficient feature reuse.
- Dense Upsampling: A significant contribution is the use of dense blocks in the "expanding" (upsampling) part of the network. This helps recover spatial resolution more effectively than standard up-convolutions.
- Unsupervised Learning: The model is trained using an image reconstruction loss. It predicts the flow, "warps" the second frame back to the first, and measures the photometric error using a robust Charbonnier penalty function. Accurate flow results in a warped image that closely matches the original first frame.
Evaluation
The model was evaluated on three standard optical flow benchmarks: Flying Chairs (synthetic), MPI Sintel (animated film with complex motion), and KITTI Optical Flow 2012 (real-world driving data).
The DenseNet approach outperformed popular architectures like VGG16, ResNet18, and the original UnsupFlowNet in unsupervised settings. On the Sintel benchmark, it achieved an Average End-Point Error (EPE) of 10.07, a marked improvement over the previous state-of-the-art of 11.19. Visually, the model produces smoother flow fields and preserves high-frequency details, such as the limbs of moving figures, which other models often blur.
Key Insights
- Implicit Deep Supervision: Direct paths from every layer to the loss function facilitate the training of deep architectures.
- Parameter Efficiency: The model uses only 2 million parameters—a massive reduction from FlowNetS's 38 million—while achieving higher accuracy.
- Feature Reuse: Access to preceding feature maps makes the model compact and robust against overfitting.
Limitations and Future Work
While promising, the model still lags behind supervised methods. Additionally, DenseNets can be memory-intensive due to frequent feature map concatenations. Future research will focus on handling large displacements and finer motion boundaries, potentially through operators like correlation layers.
Summary updated