TL;DR
This paper introduces Any-Stereo, a novel module for iterative stereo matching that enables arbitrary-scale disparity estimation. By leveraging implicit neural representations and novel feature alignment strategies, Any-Stereo significantly improves the recovery of fine details and achieves state-of-the-art performance while maintaining efficiency.
What Problem Does This Paper Address?
Estimating depth from stereo images is crucial for many computer vision applications. Traditional iterative stereo matching methods often downsample feature maps to reduce computational cost, leading to a loss of high-frequency information. While existing methods attempt to recover this lost detail during upsampling, they struggle with recovering fine structures and are limited to fixed output resolutions. This paper addresses the challenge of efficiently and accurately recovering full-resolution disparity maps at arbitrary scales, overcoming the limitations of convolutional upsampling in iterative stereo pipelines.
Methods
The core of Any-Stereo is an Implicit Neighbor Mask Function (INMF), which models the disparity map as a continuous representation learned by a Multi-Layer Perceptron (MLP). This allows for arbitrary resolution outputs without relying on fixed convolutional kernels. To further enhance the representation, two novel strategies are introduced:
- Intra-scale Similarity Unfolding (ISU): This technique enriches local information within latent codes by computing self-similarity between a pixel and its neighbors. Unlike traditional feature unfolding, ISU uses a scalar similarity to reduce computational cost.
- Cross-scale Feature Alignment (CFA): This component addresses the loss of context information during downsampling. It uses a Spatial Downsampling Block (SDB) to extract multi-scale feature maps from the input image and aligns them with the latent codes, providing richer contextual information for the INMF.
Any-Stereo is designed to seamlessly replace the upsampling module in existing iterative stereo matching frameworks.
Datasets
The paper uses the following datasets:
- SceneFlow: A large synthetic dataset used for training and pre-training models.
- KITTI (KITTI12 and KITTI15): Real-world outdoor datasets for evaluating performance on driving scenes.
- Middlebury: A real-world indoor dataset with multiple resolutions, used for evaluation.
Key Insights and Ideas
- Implicit Neural Representations for Disparity: Applying INR to disparity estimation, treating it as a continuous signal, unlocks the potential for arbitrary-scale outputs and fine detail recovery, moving beyond fixed-resolution convolutional approaches.
- Balancing Detail and Efficiency: The ISU and CFA modules demonstrate a clever trade-off between capturing rich local and multi-scale contextual information and maintaining computational efficiency, crucial for iterative methods.
- Arbitrary Scale as a Feature, Not a Limitation: The work reframes arbitrary scale upsampling from a post-processing step to an integral part of the stereo matching pipeline, enabling more robust performance across different input resolutions.
Key Results
- State-of-the-Art Performance: When integrated into existing iterative stereo models like RAFT-Stereo and IGEV-Stereo, Any-Stereo achieves new state-of-the-art results on KITTI15 and KITTI12 benchmarks, particularly in recovering fine details (e.g., improved D1-fg on KITTI15).
- Robustness to Downsampling: The proposed method significantly stabilizes performance degradation when input images are downsampled, showing nearly 50% improvement compared to baselines at 25% scale. Any-RAFT and Any-IGEV showed no degradation up to 75% downsampling on KITTI12.
- Efficiency: Despite enhancing performance, Any-Stereo maintains competitive runtime and parameter counts compared to baseline models, and in some cases, achieves better results with fewer parameters.
- Qualitative Improvements: Visualizations show Any-Stereo excels at recovering thin structures and sharp edges, outperforming traditional super-resolution methods like bicubic interpolation and LIIF, and even the original iterative baselines.
Limitations and Future Work
- Computational Cost of INR: While more efficient than some alternatives, the implicit neural representation still incurs computational cost, especially for very high resolutions or complex scene representations.
- Future Work: The paper suggests that Any-Stereo can be seamlessly integrated into various iterative stereo models, implying its broad applicability. Further exploration of different implicit neural representation architectures and their integration with other stereo matching paradigms could be future directions. The authors also highlight the potential for deployment on mobile devices due to the efficiency gains.
Summary updated