DenseNet for dense flow

Yi Zhu, Yi Zhu 0001, Shawn D. Newsam

TL;DR

This paper introduces a fully convolutional DenseNet architecture designed for unsupervised optical flow estimation. By replacing standard convolutions with dense blocks in both the contracting and expanding parts of the network, the authors achieve state-of-the-art unsupervised performance with a model that is significantly more parameter-efficient than its predecessors.

What Problem Does This Paper Address?

Optical flow—the task of estimating per-pixel motion between two consecutive video frames—is a cornerstone of computer vision, essential for everything from autonomous driving to video editing. While classical methods are often too slow for real-time use, deep learning approaches (like the pioneering FlowNet) have shifted the field toward speed and accuracy.

However, most high-performing CNNs rely on supervised learning, requiring massive synthetic datasets with "ground truth" motion labels that don't always translate well to the real world. Furthermore, standard architectures often lose fine image details as data passes through deep layers. The authors tackle two main challenges:

  1. How to build a network that preserves high-frequency details (like sharp edges) for better motion boundaries.
  2. How to train this network effectively without any human-labeled ground truth.

Methods

The authors propose a Fully Convolutional DenseNet. The architecture follows a "contracting-expanding" (encoder-decoder) structure, but with a twist:

  • Dense Blocks: Instead of traditional convolutional layers, they use Dense Blocks where each layer is connected to every other layer within the block. This ensures "maximum information flow" and allows the network to reuse features efficiently.
  • Dense Upsampling: A key contribution is replacing the standard convolutions in the "expanding" (upsampling) part of the network with dense blocks. This helps recover spatial resolution more effectively.
  • Unsupervised Learning: Instead of comparing predicted flow to a label, the model uses an image reconstruction loss. It predicts the flow, uses that flow to "warp" the second frame back to the first, and then measures the difference (photometric error) using a robust Charbonnier penalty function. If the flow is accurate, the warped image should look exactly like the original first frame.

Datasets

The model was evaluated on three standard optical flow benchmarks:

  • Flying Chairs: A large synthetic dataset used for initial pre-training.
  • MPI Sintel: A challenging dataset derived from an animated short film, featuring complex motions and realistic lighting.
  • KITTI Optical Flow 2012: Real-world data captured from a driving platform, characterized by sparse ground truth and real-world sensor noise.

Key Insights and Ideas

  • Implicit Deep Supervision: Because every layer in a dense block has a direct path to the subsequent layers and eventually the loss function, the network receives "implicit" supervision throughout. This makes it easier to train deep architectures.
  • Parameter Efficiency: The DenseNet model uses only 2 million parameters, a staggering reduction compared to the 38 million parameters in FlowNetS, yet it achieves superior accuracy.
  • Feature Reuse: By allowing layers to access feature maps from all preceding layers, the model is less prone to overfitting and more compact.
  • The "Aha" Moment in Upsampling: The authors found that while many models use simple skip connections to bring back detail, actually performing dense computations during the upsampling phase is what truly captures the fine-grained motion of human boundaries and car shapes.

Key Results

The proposed model outperformed other popular architectures (VGG16, ResNet18, and the original UnsupFlowNet) across the board in an unsupervised setting:

  • Accuracy: On the Sintel benchmark, the DenseNet approach achieved an Average End-Point Error (EPE) of 10.07, compared to 11.19 for the previous state-of-the-art unsupervised method.
  • Qualitative Improvements: Visual results show that the DenseNet produces much smoother flow fields and preserves high-frequency details (like the limbs of a moving person) that other unsupervised models often blur or miss.
  • Robustness: The model proved remarkably robust to overfitting, performing well on Sintel even when trained from scratch without pre-training on Flying Chairs.

Limitations and Future Work

Despite the improvements, the authors note a few hurdles:

  • Supervision Gap: While the model closes the gap, it still lags behind supervised methods that have access to ground-truth labels.
  • Memory Bandwidth: DenseNets can be memory-intensive due to the high number of feature map concatenations, which might limit their use in some real-time embedded systems.
  • Complex Motion: Future work is needed to better handle very large displacements and even finer motion boundaries. The authors suggest exploring novel operators, such as correlation layers, to further improve the network's ability to match features between frames.

Summary updated

Abstract

Classical approaches for estimating optical flow have achieved rapid progress in the last decade. However, most of them are too slow to be applied in real-time video analysis. Due to the great success of deep learning, recent work has focused on using CNNs to solve such dense prediction problems. In this paper, we investigate a new deep architecture, Densely Connected Convolutional Networks (DenseNet), to learn optical flow. This specific architecture is ideal for the problem at hand as it provides shortcut connections throughout the network, which leads to implicit deep supervision. We extend current DenseNet to a fully convolutional network to learn motion estimation in an unsupervised manner. Evaluation results on three standard benchmarks demonstrate that DenseNet is a better fit than other widely adopted CNN architectures for optical flow estimation.
BibTeX
@inproceedings{zhu2017densenet,
  author = {Yi Zhu and Yi Zhu and Shawn D. Newsam},
  title = {{DenseNet for dense flow}},
  booktitle = {ICIP},
  year = {2017},
  doi = {10.1109/icip.2017.8296389},
}

Loading related content...

Figure