Revealing Scenes by Inverting Structure From Motion Reconstructions

Francesco Pittaluga, Sanjeev J. Koppal, Sing Bing Kang, Sudipta N. Sinha

Abstract

Many 3D vision systems localize cameras within a scene using 3D point clouds. Such point clouds are often obtained using structure from motion (SfM), after which the images are discarded to preserve privacy. In this paper, we show, for the first time, that such point clouds retain enough information to reveal scene appearance and compromise privacy. We present a privacy attack that reconstructs color images of the scene from the point cloud. Our method is based on a cascaded U-Net that takes as input, a 2D multichannel image of the points rendered from a specific viewpoint containing point depth and optionally color and SIFT descriptors and outputs a color image of the scene from that viewpoint. Unlike previous feature inversion methods, we deal with highly sparse and irregular 2D point distributions and inputs where many point attributes are missing, namely keypoint orientation and scale, the descriptor image source and the 3D point visibility. We evaluate our attack algorithm on public datasets and analyze the significance of the point cloud attributes. Finally, we show that novel views can also be generated thereby enabling compelling virtual tours of the underlying scene.

The Privacy Risk of 3D Mapping

Modern Augmented Reality (AR) and autonomous systems, such as ARKit, ARCore, and HoloLens, rely on Structure from Motion (SfM) to navigate. SfM creates a 3D map of an environment consisting of a "sparse point cloud"—a collection of thousands of individual 3D points, each associated with mathematical descriptors (like SIFT) that help the camera recognize its location.

To protect user privacy, these systems typically discard the original photos and only store the sparse point cloud. The industry assumption has been that these abstract clouds of dots are uninterpretable by humans. This research challenges that assumption, demonstrating that these mathematical representations can be "inverted" to reveal sensitive environments like homes and offices.

The Reconstruction Pipeline

The proposed privacy attack pipeline consists of three specialized neural networks arranged in a cascade to handle the sparse and irregular nature of SfM data:

  1. VisibNet (Visibility Estimation): Sparse point clouds often contain points from the background that should be occluded by foreground objects. VisibNet predicts which points are actually visible from a specific viewpoint, preventing "ghosting" artifacts where background textures leak through foreground walls.
  2. CoarseNet (Image Reconstruction): This network takes a 2D projection of the visible points—including their depth, color, and SIFT descriptors—and generates a rough RGB image. It uses a U-Net architecture to interpolate the massive gaps between the sparse points.
  3. RefineNet (Detail & Adversarial Refinement): Using a Generative Adversarial Network (GAN) framework, this stage sharpens the coarse image, adding fine details and plausible textures to make the output look like a real photograph.
The cascaded neural network architecture: VisibNet estimates visibility, CoarseNet generates a rough RGB image, and RefineNet sharpens the final output.

Experimental Evaluation

The models were trained and evaluated on two major datasets to ensure the attack works in diverse environments:

  • MegaDepth (MD): A large-scale dataset of approximately 150,000 internet photos of 196 landmark scenes (mostly outdoors).
  • NYU Depth V2: A dataset of approximately 400,000 indoor images captured with a Kinect, representing the types of sensitive home/office environments where AR devices are frequently used.

The researchers used COLMAP, a standard SfM software, to generate the "ground truth" 3D point clouds used for training the inversion models.

Key Results and Privacy Implications

The attack demonstrates that privacy is often an illusion in 3D mapping. The research highlights several critical findings:

  • High Fidelity: Reconstructed images closely resemble the source images across both outdoor landmarks and indoor rooms. On the NYU dataset, the system achieved a mean classification accuracy of over 90% for visibility estimation.
  • SIFT as a Visual Source: One of the most surprising findings is that SIFT descriptors—mathematical vectors intended only for computer matching—retain a massive amount of visual information. Even without any color data, the network can reconstruct recognizable scenes just from these descriptors.
  • Resilience to Sparsity: The attack remains effective even when 80% of the points are removed. While the Mean Absolute Error (MAE) increases slightly, the scenes remain highly recognizable.
  • Color Recovery: Even when the input point cloud contains zero color information, RefineNet is able to synthesize plausible colors for common objects (like wooden bookshelves or white walls), making the attack even more visually revealing.
Qualitative results showing the progression from input point clouds (top rows) to reconstructed images (middle rows) compared to ground truth (bottom rows) for various scenes.

Conclusion and Future Work

This work proves that "discarding images to preserve privacy" is a flawed strategy if the resulting 3D map is stored in a standard format. The map essentially is the image, just in a different form. Furthermore, because the attack works on a 3D model, an attacker can generate "virtual tours" of a room from angles the camera never even visited.

The authors suggest this work should spark a search for "privacy-preserving" 3D representations—ways to store map data that allow a robot to localize but prevent a human (or a GAN) from reconstructing the scene. Potential defenses include obfuscating the map geometry or developing new descriptors that do not retain human-interpretable visual information.

Summary updated

Abstract

Many 3D vision systems localize cameras within a scene using 3D point clouds. Such point clouds are often obtained using structure from motion (SfM), after which the images are discarded to preserve privacy. In this paper, we show, for the first time, that such point clouds retain enough information to reveal scene appearance and compromise privacy. We present a privacy attack that reconstructs color images of the scene from the point cloud. Our method is based on a cascaded U-Net that takes as input, a 2D multichannel image of the points rendered from a specific viewpoint containing point depth and optionally color and SIFT descriptors and outputs a color image of the scene from that viewpoint. Unlike previous feature inversion methods, we deal with highly sparse and irregular 2D point distributions and inputs where many point attributes are missing, namely keypoint orientation and scale, the descriptor image source and the 3D point visibility. We evaluate our attack algorithm on public datasets and analyze the significance of the point cloud attributes. Finally, we show that novel views can also be generated thereby enabling compelling virtual tours of the underlying scene.
BibTeX
@inproceedings{pittaluga2019revealing,
  author = {Francesco Pittaluga and Sanjeev J. Koppal and Sing Bing Kang and Sudipta N. Sinha},
  title = {{Revealing Scenes by Inverting Structure From Motion Reconstructions}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2019},
  url = {https://openreview.net/forum?id=F2gdvbzyjl},
}
Figure