Presented at ICLR 2019, DPSNet (Deep Plane Sweep Network) is a framework that successfully marries traditional geometric computer vision with modern deep learning to solve the problem of multiview stereo (MVS). By mimicking traditional geometric pipelines—feature extraction, cost volume construction via warping, and cost aggregation—within a neural network, it achieves state-of-the-art depth reconstruction even in challenging textureless or reflective regions.
The Challenge of Multiview Stereo
Multiview stereo aims to reconstruct the 3D geometry of a scene from a collection of 2D images taken from different viewpoints. Traditionally, this relies on "photo-consistency"—the idea that a 3D point should look the same in every image. However, traditional methods often fail in "textureless" areas (like a blank white wall) or "reflective" surfaces (like a window), where pixel-to-pixel matching is highly ambiguous.
While Convolutional Neural Networks (CNNs) can use semantic context to infer depth in these tricky areas, early deep MVS attempts were often not end-to-end. They either required pre-computed cost volumes as input or were restricted to simple binocular (two-camera) setups. The authors of DPSNet sought to create a system that could be trained directly on raw images and camera poses to produce high-quality dense depth maps for any number of input views.
The DPSNet Architecture
The architecture of DPSNet follows a pipeline inspired by classical geometry, reformulated as differentiable layers:
- Multi-scale Feature Extraction: The network uses a Spatial Pyramid Pooling (SPP) module to extract features. This allows the model to consider both local details and global context (e.g., recognizing that a large, uniform blue area is likely a sky).
- Differentiable Plane Sweep: This is the core innovation. The network "sweeps" through a series of virtual depth planes. For each plane, it warps the features from the target images onto the reference image's viewpoint using a differentiable warping module, similar to a Spatial Transformer Network.
- Cost Volume Generation: Instead of using a simple mathematical difference (like absolute difference) to compare pixels, DPSNet concatenates the reference features with the warped target features. This allows the subsequent layers to learn the best matching metric, capturing complex, non-linear ways to identify matches that simple subtraction misses.
- Context-Aware Cost Aggregation: To clean up noise in the cost volume, the authors introduce a sub-network that uses the reference image as a guide. It applies dilated 2D convolutions to refine each "depth slice," ensuring that depth boundaries align tightly with visual boundaries in the image.
- Depth Regression: Finally, the network uses a "soft argmin" operation (a softmax followed by a weighted sum) to regress a continuous depth value for every pixel.
Key Results and Insights
DPSNet was trained and evaluated on several standard benchmarks, including SUN3D, RGBD, MVS (DeMoN datasets), Scenes11, and ETH3D. It consistently outperformed both traditional methods (like COLMAP) and contemporary deep learning models (like DeMoN and DeepMVS).
- Accuracy: On the MVS dataset, DPSNet achieved an Absolute Relative Error (Abs Rel) of 0.0722, significantly lower than DeepMVS (0.2305).
- Completeness: On the ETH3D benchmark, DPSNet achieved 100% completeness—providing a depth estimate for every single pixel—while maintaining a lower geometric error than filtered versions of traditional algorithms, which often discard up to 30% of pixels as "unreliable."
- Generalization: The model showed strong performance on ETH3D even though it was never trained on that specific data, proving the robustness of its learned geometric features.
The most powerful insight from DPSNet is the concept of Geometry as a Layer. Traditional geometric algorithms, like plane sweeping, do not need to be replaced by black-box AI. Instead, they can be integrated as differentiable modules, allowing the network to learn around the physics of light and perspective. Furthermore, by feeding the original reference image back into the cost aggregation step, the network learns to smooth depth estimates within objects while keeping edges sharp at boundaries.
Limitations and Future Work
While highly effective, DPSNet has a few limitations that pave the way for future research:
- Camera Poses: The system currently requires pre-calibrated camera intrinsics and extrinsics. Future versions could potentially estimate these poses internally.
- Viewpoint Selection: The current model averages cost volumes from all available views. In real-world scenarios with dozens of images, an intelligent "view selection" mechanism to weigh the most relevant views would likely improve results.
- Semantic Integration: While the network utilizes visual context, it does not explicitly perform semantic segmentation. Integrating instance-level semantics (e.g., explicitly recognizing object shapes) is a promising future direction.
Summary updated