Abstract
Occlusion areas—pixels visible in only one of two consecutive frames—pose a significant challenge for computer vision tasks such as optical flow estimation and frame interpolation. Traditional detection methods often rely on slow energy function optimizations. This paper proposes a lightweight neural network for occlusion detection based on the PWC-net architecture. By utilizing a multi-scale feature pyramid and a dedicated occlusion estimation sub-network, the model achieves high accuracy while maintaining computational efficiency. Experimental results on the MPI-Sintel dataset show that the proposed method outperforms several complex analogs and provides a viable alternative to heavy-duty models.
Introduction
In video processing, "occlusions" (opening or closing areas) occur when pixels appear or disappear between frames. Accurate detection of these regions is critical; without it, algorithms like optical flow estimation attempt to find matches for non-existent pixels, leading to significant noise and artifacts. However, occlusion detection faces a "chicken-and-egg" problem: high-quality motion information is needed to find occlusions, but knowing occlusions helps improve motion estimation by avoiding false pixel correspondences.
Most existing solutions are either computationally prohibitive or fail to capture the motion-based nature of occlusions. This research introduces an efficient method that integrates occlusion estimation directly into a pyramid-based motion estimation framework.
Methodology
The proposed architecture is built upon PWC-net, which employs a Pyramid of features, Warping, and a Cost volume. The core innovation lies in the addition of a dedicated occlusion estimation sub-network at every level of the feature pyramid.
Architecture and Training
Each occlusion sub-network consists of six convolutional layers with depths ranging from 128 down to 2. To preserve spatial detail, the authors utilized dense connections (concatenation), where the output of each layer is passed to all subsequent layers. This design allows the network to retain fine-grained information from earlier layers while learning complex features.
The model was trained on the FlyingChairs2 and FlyingThings3D synthetic datasets, which provide ground truth for motion and occlusions. A multi-scale binary cross-entropy loss was applied at each pyramid level, forcing the model to learn occlusion patterns at various resolutions—from coarse object movements to sharp boundaries.
Experimental Results
The method was evaluated on the MPI-Sintel (Final) benchmark and compared against 15 other algorithms.
Performance and Efficiency
The proposed model demonstrates a significant improvement in efficiency over state-of-the-art heavy models like FlowNet 2.0:
- Parameters: 23 million vs. 117 million (5x reduction).
- Training Time: 3 days vs. 12 days (4x faster).
- Accuracy: The model achieved an F-measure of 0.531, significantly outperforming the baseline PWC-net (0.479).
As shown in the Precision-Recall curve, the proposed method consistently maintains higher precision across various recall levels compared to the baseline, validating the effectiveness of the integrated occlusion sub-networks.
Conclusion
By piggybacking on the efficient feature extraction of PWC-net, the authors developed a high-performance occlusion detection model that is both lightweight and accurate. While it faces challenges with extremely thin occlusions (1–2 pixels) due to resolution constraints, its balance of speed and precision makes it highly suitable for real-time video processing applications. Future work will focus on expanding the temporal window to further improve detection in complex scenes.
Summary updated