Abstract
Significant progress has been made for estimating optical flow using deep neural networks. Advanced deep models achieve accurate flow estimation often with a considerable computation complexity and time-consuming training processes. In this work, we present a lightweight yet effective model for real-time optical flow estimation, termed FDFlowNet (fast deep flownet). We achieve better or similar accuracy on the challenging KITTI and Sintel benchmarks while being about 2 times faster than PWC-Net. This is achieved by a carefully-designed structure and newly proposed components. We first introduce an U-shape network for constructing multi-scale feature which benefits upper levels with global receptive field compared with pyramid network. In each scale, a partial fully connected structure with dilated convolution is proposed for flow estimation that obtains a good balance among speed, accuracy and number of parameters compared with sequential connected and dense connected structures. Experiments demonstrate that our model achieves state-of-the-art performance while being fast and lightweight.
Introduction
Optical flow estimation—the task of tracking pixel-level motion between consecutive frames—is fundamental to computer vision applications like autonomous driving and video stabilization. However, existing models often struggle to balance high accuracy with real-time performance. Large models like FlowNet2 are accurate but computationally prohibitive, while lightweight pyramid-based models like PWC-Net can lose semantic context at higher resolutions. FDFlowNet addresses this "Goldilocks" problem by rethinking feature extraction and motion estimation to achieve a 2x speedup over PWC-Net without sacrificing accuracy.
Architecture
FDFlowNet introduces two primary architectural innovations designed to optimize the trade-off between speed and precision.
U-shape Network
Traditional coarse-to-fine models use a simple downward feature pyramid. FDFlowNet replaces this with a symmetrical U-shape network. This structure first downsamples the image to capture global context and then "up-fuses" this high-level semantic information back into high-resolution layers. This ensures that the flow estimator maintains a "big picture" understanding of the scene even when refining fine details.
Partial Fully Connected (PFC) Flow Estimator
The flow estimation at each scale is handled by a novel Partial Fully Connected (PFC) structure. While sequential connections are fast but prone to information loss, and dense connections are accurate but slow, the PFC structure offers a hybrid approach. By using a selective connectivity pattern and incorporating dilated convolutions, the PFC structure increases the receptive field—allowing the model to "see" more of the image—without adding significant parameter overhead.
Experiments and Results
The model was trained on the FlyingChairs and FlyingThings3D datasets and evaluated on the MPI Sintel and KITTI (2012 & 2015) benchmarks.
Performance and Efficiency
FDFlowNet demonstrates state-of-the-art efficiency:
- Speed: It processes frames in approximately 16.7ms (~60 FPS) on an NVIDIA 1080Ti, making it 2x faster than PWC-Net and 3.2x faster than LiteFlowNet.
- Accuracy: On Sintel Clean, FDFlowNet achieved an End-Point Error (EPE) of 3.71, surpassing PWC-Net's 4.39. It also reached a state-of-the-art EPE of 1.5 on KITTI 2012.
- Compactness: With only 5.79 million parameters, the model is highly suitable for edge device deployment.
Conclusion
FDFlowNet successfully bridges the gap between lightweight design and high-performance optical flow estimation. By leveraging a U-shape feature extractor and a PFC estimator with dilated convolutions, it provides a practical solution for real-time applications. Future work may focus on improving robustness to extreme occlusions and further optimizing performance for specialized mobile hardware.
Summary updated