Revealing Scenes by Inverting Structure from Motion Reconstructions

Francesco Pittaluga, Sanjeev J. Koppal, Sing Bing Kang, Sudipta N. Sinha

Abstract: Many 3D vision systems localize cameras within a scene using 3D point clouds. Such point clouds are often obtained using structure from motion (SfM), after which the images are discarded to preserve privacy. In this paper, we show, for the first time, that such point clouds retain enough information to reveal scene appearance and compromise privacy. We present a privacy attack that reconstructs color images of the scene from the point cloud. Our method is based on a cascaded U-Net that takes as input, a 2D multichannel image of the points rendered from a specific viewpoint containing point depth and optionally color and SIFT descriptors and outputs a color image of the scene from that viewpoint. Unlike previous feature inversion methods, we deal with highly sparse and irregular 2D point distributions and inputs where many point attributes are missing, namely keypoint orientation and scale, the descriptor image source and the 3D point visibility. We evaluate our attack algorithm on public datasets and analyze the significance of the point cloud attributes. Finally, we show that novel views can also be generated thereby enabling compelling virtual tours of the underlying scene.

Introduction

Modern Augmented Reality (AR) and autonomous systems, such as ARCore, ARKit, and HoloLens, rely on Structure from Motion (SfM) to map their surroundings. These systems generate 3D maps composed of sparse point clouds. To protect user privacy, the original images used to construct these maps are typically discarded, based on the assumption that sparse point clouds are "uninterpretable" to humans and thus effectively anonymized.

This research challenges that assumption by addressing the "feature inversion" problem. The authors demonstrate that it is possible to take a sparse, irregular 2D projection of a 3D point cloud and reconstruct a detailed, recognizable color image. This attack is particularly potent because it succeeds even when critical information—such as point visibility, scale, and orientation—is absent from the stored data.

Methodology: The Cascaded U-Net Architecture

The core of the proposed privacy attack is a three-stage pipeline based on a cascaded U-Net architecture. This design is specifically engineered to handle the challenges of highly sparse and irregular data.

The cascaded architecture consisting of VisibNet, CoarseNet, and RefineNet.
  1. VisibNet (Visibility Estimation): Sparse point clouds do not inherently indicate which points are occluded from a specific viewpoint. VisibNet is a sub-network trained to classify each projected point as "visible" or "occluded." This step is crucial for preventing "ghosting" artifacts where background points erroneously appear through foreground objects.
  2. CoarseNet (Image Reconstruction): This network receives the filtered 2D points (which may include depth, color, or SIFT descriptors) and generates a coarse RGB image. It utilizes encoder-decoder layers with skip connections to "hallucinate" the missing information between sparse points.
  3. RefineNet (Adversarial Refinement): To improve visual quality, the authors employ a Generative Adversarial Network (GAN) framework. RefineNet processes the output of CoarseNet to add fine-grained details, textures, and plausible colors, resulting in a final image that closely resembles the original scene.

Experimental Evaluation

The model was trained and evaluated on two large-scale datasets: MegaDepth (MD), containing approximately 150,000 outdoor landmark images, and NYU2, featuring 400,000 indoor images. Standard SfM software (COLMAP) was used to generate the ground truth 3D point clouds and visibility labels for training.

Key Results

The results demonstrate a significant privacy vulnerability. Even "minimalist" representations, such as those storing only SIFT descriptors without color, are susceptible to reconstruction. The descriptors contain sufficient semantic information for the neural network to accurately predict the colors and textures of the scene.

Qualitative results showing point cloud inputs (top), reconstructed images (middle), and ground truth (bottom) for various scenes.
  • Visual Fidelity: Reconstructed images maintain the layout, object placement, and color of the originals. The visibility classifier achieved over 93.8% accuracy on MegaDepth.
  • Quantitative Performance: On the NYU dataset, using all attributes (depth, color, SIFT) yielded a Structural Similarity (SSIM) index of 0.594, which is remarkably high given the sparse input.
  • Robustness: The system remains effective even when 80% of the points are removed from an already sparse cloud. Furthermore, the network can reconstruct recognizable scenes using depth information alone.
  • Novel View Synthesis: Since the attack operates on 3D data, an attacker can generate "virtual tours" of a private space from viewpoints never captured by the original camera.

Conclusion and Future Work

This work reveals that current methods for "anonymizing" 3D maps by discarding images are insufficient. While the system has limitations—such as difficulty with moving objects or complex occlusions in very dense scenes—the ability to reconstruct recognizable private environments from sparse points is a major security concern. This research highlights the urgent need for "privacy-preserving" 3D representations that allow for localization without exposing sensitive visual data.

Summary updated

Abstract

Many 3D vision systems localize cameras within a scene using 3D point clouds. Such point clouds are often obtained using structure from motion (SfM), after which the images are discarded to preserve privacy. In this paper, we show, for the first time, that such point clouds retain enough information to reveal scene appearance and compromise privacy. We present a privacy attack that reconstructs color images of the scene from the point cloud. Our method is based on a cascaded U-Net that takes as input, a 2D multichannel image of the points rendered from a specific viewpoint containing point depth and optionally color and SIFT descriptors and outputs a color image of the scene from that viewpoint. Unlike previous feature inversion methods, we deal with highly sparse and irregular 2D point distributions and inputs where many point attributes are missing, namely keypoint orientation and scale, the descriptor image source and the 3D point visibility. We evaluate our attack algorithm on public datasets and analyze the significance of the point cloud attributes. Finally, we show that novel views can also be generated thereby enabling compelling virtual tours of the underlying scene.
BibTeX
@inproceedings{pittaluga2019revealing,
  author = {Francesco Pittaluga and Sanjeev J. Koppal and Sing Bing Kang and Sudipta N. Sinha},
  title = {{Revealing Scenes by Inverting Structure from Motion Reconstructions}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2019},
  url = {https://openreview.net/forum?id=F2gdvbzyjl},
}
Figure