TL;DR
This paper introduces 3D-LatentMapper, a novel framework for single-view 3D shape reconstruction that operates within the latent space of a pre-trained generative model. By leveraging features from Vision Transformers (ViT) and CLIP, it achieves view-agnostic reconstruction, even with occlusions, and significantly outperforms existing methods in both quality and speed.
What Problem Does This Paper Address?
Reconstructing 3D shapes from a single 2D image (Single-View Reconstruction or SVR) is a long-standing challenge in computer vision. Unlike 2D image processing, 3D data has diverse and often problematic representations (e.g., point clouds, voxels, meshes), making it difficult to apply standard deep learning techniques. Existing methods often struggle with occlusions, require specific viewpoints, or are computationally expensive due to complex optimization or mesh generation processes. This limits their utility in interactive applications like 3D design.
Methods
The core of 3D-LatentMapper is a novel mapping network that bridges the gap between image features and the latent space of a pre-trained 3D generative model, specifically Deep Implicit Templates (DIT). The framework leverages:
- Vision Transformer (ViT) Key Descriptors: Instead of standard CNN features, the authors extract "key descriptors" from intermediate layers of a pre-trained ViT. These descriptors are found to be robust to pose and scale variations and capture rich semantic information. They are further processed using a bag-of-descriptors approach with K-means clustering.
- CLIP Embeddings: The paper utilizes CLIP, a joint image-text model, to extract dense image features. CLIP's ability to align visual and textual concepts is crucial for generating semantically consistent reconstructions, even for objects not explicitly seen during training.
- Autoencoder for ViT Features: A small autoencoder is used to compress the ViT descriptors into a more manageable dimensionality (512D), creating a dense feature representation.
- Mapping Network: The compressed ViT features and CLIP embeddings are concatenated and fed into a feed-forward network. This network learns to map these combined features to the latent space of the DIT model.
- Real-to-Synthetic Image Translation: To handle real-world images, a CycleGAN-based module is employed to translate real images into a synthetic domain that the main reconstruction framework can process.
Datasets
- ShapeNetV2: Used for training and evaluating the core SVR framework, specifically the car and airplane categories.
- Car Connection Dataset: A large, publicly available dataset of car images used for training the real-to-synthetic image translation module. This dataset was filtered using CLIP and sentence templates to create a cleaner subset.
Key Insights and Ideas
- ViT Key Descriptors as Powerful Features: The paper highlights that intermediate ViT "key descriptors" are surprisingly effective for 3D reconstruction, outperforming traditional CNN features by capturing richer, more robust semantic and positional information.
- CLIP for Semantic Robustness: Using CLIP embeddings allows the model to generalize better, inferring missing parts or understanding object semantics even when the input image is incomplete or ambiguous.
- Latent Space Manipulation is Key: By operating entirely within the latent space of a generative model (DIT), the framework avoids the computationally expensive and non-differentiable steps of explicit mesh generation during inference, leading to faster and more flexible reconstruction.
- View-Agnostic Reconstruction: The framework's ability to reconstruct shapes regardless of the input viewpoint is a significant advantage over many prior methods.
Key Results
- Qualitative: 3D-LatentMapper produces high-fidelity 3D reconstructions that capture fine details and even infer plausible unseen features. It outperforms AtlasNet and IM-NET in visual quality.
- Quantitative: The method achieves state-of-the-art performance on Chamfer Distance (CD) and Earth Mover's Distance (EMD) metrics for both car and airplane categories, significantly outperforming AtlasNet and IM-NET.
- Speed: The framework is remarkably fast, with an inference time of 8.030 seconds per shape, significantly faster than IM-NET (13.280 seconds) and competitive with AtlasNet (0.112 seconds), while offering superior quality.
- Real Image Reconstruction: The real-to-synthetic translation module enables reasonable reconstruction from real-world images, even if the synthetic images are not perfect.
Limitations and Future Work
- Base Generative Model Dependency: The quality of reconstruction is inherently limited by the expressiveness and richness of the latent space of the base 3D generative model (DIT). A more capable base model could lead to even better results.
- Real-to-Synthetic Translation Imperfections: While functional, the real-to-synthetic image translation module can produce artifacts, which can impact the final reconstruction quality, especially for challenging real-world images.
- Future Work: The authors suggest that using a base model with a richer latent space would be a promising direction. Further improvements to the real-to-synthetic translation module could also enhance performance on real-world data.
Summary updated