Deep vanishing point detection: Geometric priors make dataset variations vanish

Yancong Lin, Ruben Wiersma, Silvia Laura Pintea, Klaus Hildebrandt, Elmar Eisemann, Jan van Gemert

TL;DR

This paper introduces a novel deep learning approach for vanishing point detection that significantly improves data efficiency and robustness by incorporating geometric priors. By mapping image features to the Hough Transform and then to a Gaussian sphere, the model leverages inherent geometric knowledge, reducing reliance on massive annotated datasets and enabling better generalization across different domains and problem variations.

What Problem Does This Paper Address?

Detecting vanishing points (VPs) in images is crucial for various computer vision tasks like camera calibration, scene understanding, and autonomous driving. While deep learning has shown great promise, current methods suffer from several drawbacks: they require extensive, expensive, and error-prone annotated datasets; demand significant computational resources for training; struggle with domain shifts (changes in data collection or environment); and often need complete architectural redesigns for minor problem variations. This paper aims to overcome these limitations by making deep VP detection models more data-efficient, robust to domain changes, and adaptable to variations like detecting a non-fixed number of VPs.

Methods

The core innovation of this work lies in injecting geometric priors into a deep learning framework for vanishing point detection. Instead of learning all geometric knowledge from scratch, the authors incorporate two specific priors:

  1. Hough Transform: Image features extracted by a CNN are mapped into the Hough Transform space. This space parameterizes lines using polar coordinates (offset and angle), effectively representing each line as a bin in a 2D histogram. This step helps in identifying individual lines.
  2. Gaussian Sphere Mapping: The Hough Transform bins, representing lines, are then projected onto a Gaussian sphere. On this sphere, image lines become great circles, and vanishing points are found at the intersections of these great circles. This mapping constrains the problem to a bounded parameter space and facilitates detection of VPs outside the image view.

Both the Hough Transform and Gaussian sphere mapping modules are designed to be end-to-end trainable, allowing the network to learn from data while benefiting from the embedded geometric knowledge. For localization on the sphere, the authors employ spherical convolutions, specifically using EdgeConv modules to process the point cloud representation of the hemisphere.

Datasets

The authors evaluate their model on several datasets:

  • SU3 (SceneCity Urban 3D): A synthetic dataset with 3 orthogonal vanishing points.
  • ScanNet: A real-world dataset with 3 orthogonal vanishing points, known for its varying image content.
  • YUD (Urban 3D): A dataset capturing urban scenes, with only 102 images, making it a challenging small-data scenario.
  • NYU Depth: A real-world dataset that does not follow the Manhattan assumption, featuring a varying number of vanishing points (1-8) per image. This dataset is crucial for testing adaptability to non-Manhattan scenarios and handling varying VP counts.

Key Insights and Ideas

  • Geometric Priors as a Shortcut: The paper demonstrates that explicitly incorporating geometric knowledge (Hough Transform and Gaussian sphere) significantly reduces the need for massive annotated datasets. This "prior knowledge" acts as a strong inductive bias, guiding the network and saving valuable annotation effort and computational resources.
  • Robustness to Domain Shifts: By relying on fundamental geometric principles rather than purely data-driven patterns, the model becomes less sensitive to variations in data collection or environmental conditions, leading to better generalization across different datasets.
  • Adaptability to Problem Variations: The interpretability of geometric priors allows for easier adaptation to slightly different problem formulations, such as detecting a variable number of vanishing points, which is a significant advantage over rigid deep learning architectures.
  • Bridging Classical and Deep Methods: The approach effectively combines the strengths of classical geometric methods (Hough Transform, Gaussian sphere) with the representational power of deep learning (CNNs, spherical convolutions), creating a more powerful and efficient system.

Key Results

  • Comparable Accuracy on Large Datasets: The proposed model achieves accuracy comparable to existing state-of-the-art models on large datasets like SU3 and ScanNet, even with significantly fewer parameters.
  • Improved Data Efficiency: On smaller subsets of datasets (e.g., 1% of ScanNet or SU3), the model shows a graceful degradation in performance, outperforming methods like NeurVPS which suffer drastic accuracy drops. This highlights the model's ability to learn effectively from limited data.
  • Robustness to Non-Manhattan Scenarios: The model performs exceptionally well on the NYU Depth dataset, which lacks the Manhattan assumption and has a varying number of VPs, outperforming baselines like CONSAC and J-Linkage.
  • Cross-Dataset Transferability: The model demonstrates strong performance when tested on datasets it wasn't explicitly trained on, indicating improved generalization and robustness to domain shifts. For instance, it outperforms CONSAC on the challenging ScanNet dataset when trained on synthetic data.
  • Speed Improvements: Using multi-scale sampling with the Manhattan assumption, the model achieves significant speedups (e.g., 23 FPS) while maintaining accuracy.

Limitations and Future Work

  • Quantization Errors: The current implementation relies on pre-computed mappings and discrete sampling (Hough histogram, Fibonacci lattice for sphere sampling). These quantization steps introduce errors, limiting accuracy, particularly in the low-error region on datasets like SU3.
  • Analytical Mapping: The authors suggest exploring an analytical mapping from image pixels to the Gaussian sphere as a future research direction to mitigate quantization errors.
  • Data Dependency: While significantly more data-efficient, the model still requires hundreds of fully labeled samples for training.
  • Unsupervised/Weakly-Supervised Learning: Investigating the application of these geometric priors in unsupervised or weakly-supervised settings is proposed as a future avenue.

Summary updated

Abstract

Deep learning has greatly improved vanishing point detection in images. Yet, deep networks require expensive annotated datasets trained on costly hardware and do not generalize to even slightly different domains and minor problem variants. Here, we address these issues by injecting deep vanishing point detection networks with prior knowledge. This prior knowledge no longer needs to be learned from data, saving valuable annotation efforts and compute, unlocking realistic few-sample scenarios, and reducing the impact of domain changes. Moreover, because priors are interpretable, it is easier to adapt deep networks to minor problem variations such as switching between Manhattan and non-Manhattan worlds. We incorporate two end-to-end trainable geometric priors: (i) Hough Transform -- mapping image pixels to straight lines, and (ii) Gaussian sphere -- mapping lines to great circles whose intersections denote vanishing points. Experimentally, we ablate our choices and show comparable accuracy as existing models in the large-data setting. We then validate that our model improves data efficiency, is robust to domain changes, and can easily be adapted to a non-Manhattan setting.
BibTeX
@inproceedings{lin2022deep,
  author = {Yancong Lin and Ruben Wiersma and Silvia Laura Pintea and Klaus Hildebrandt and Elmar Eisemann and Jan van Gemert},
  title = {{Deep vanishing point detection: Geometric priors make dataset variations vanish}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2022},
  doi = {10.1109/cvpr52688.2022.00601},
}
Figure