Abstract
Traditional 3D shape retrieval systems often operate on a global scale, matching entire objects based on overall silhouettes while overlooking the modular nature of real-world design. This paper introduces "3D Pick 'n' Mix," a novel framework that enables part-level 3D model retrieval from 2D image queries. By constructing a joint embedding space for 2D pixels and 3D part manifolds, the proposed EmbedNet architecture allows users to select specific components from different images—such as the legs of one chair and the backrest of another—to retrieve the most semantically similar 3D model from a database.
Introduction: Beyond "Black Box" Retrieval
Most existing 3D shape retrieval methods treat objects as indivisible units. When presented with a 2D image, these systems attempt to find a 3D model that matches the entire silhouette. However, this approach fails to account for the modularity inherent in design and consumer preferences. A user might admire the armrests of one furniture piece but prefer the base of another.
The challenge in part-level reasoning is twofold: global retrieval methods prioritize overall shape over fine-grained details, and the "2D-to-3D gap" makes it difficult to map occluded, textured 2D images to clean 3D geometries without extensive manual labeling.
Methodology: The EmbedNet Architecture
The core of the system is EmbedNet, a deep learning architecture designed to bridge the gap between 2D visual data and 3D part manifolds. The methodology consists of three primary stages:
- Part Manifold Construction: Separate 128-dimensional "shape spaces" are created for distinct object parts (e.g., legs, seats). Using Light Field Descriptors (LfD) and HoG pyramids, 3D parts are arranged via non-linear Multi-Dimensional Scaling (MDS) so that geometrically similar parts are clustered together.
- EmbedNet Processing: A Convolutional Neural Network (CNN) performs joint semantic segmentation, feature extraction, and coordinate regression. It identifies part-specific pixels, focuses on their geometry by combining RGB data with part heatmaps, and maps them directly onto the pre-constructed 3D manifolds.
- Cross-Embedding Optimization: During retrieval, the system takes coordinates from multiple user-selected parts and optimizes to find the single 3D model in the database that best satisfies all constraints simultaneously.
Datasets and Evaluation
The researchers utilized ShapeNet as the primary source for 3D models and part annotations. To evaluate the system, they employed the standard ExactMatch dataset for whole-object retrieval and introduced a new benchmark, ExactPartMatch, specifically for part-level accuracy. For real-world robustness, Pascal 3D and IKEA Images were used to provide background augmentation during training, ensuring the network can handle "in the wild" photography.
Key Insights and Results
A critical finding was that "smoothing" the manifold by removing high-frequency HoG components improved generalization, allowing the network to better handle novel shapes. Furthermore, training on synthetic renders with real-world background augmentation proved highly effective for processing actual internet photos.
Performance Highlights: * Part Retrieval Superiority: On the ExactPartMatch dataset, the proposed method achieved a top-5 retrieval accuracy of 76%, significantly outperforming traditional methods (65%). * Global Competitiveness: The system remains competitive with state-of-the-art whole-object retrieval models on the ExactMatch benchmark. * Robustness: The joint learning of segmentation and regression allows the system to handle occlusions and difficult viewing angles that typically cause global methods to fail.
Conclusion and Future Work
While the current system is a powerful search engine for existing models, it faces challenges with perspective ambiguity where 2D images lack sufficient geometric information. Future research may transition from retrieval to generative modeling, potentially "stitching" together new 3D meshes from identified parts rather than just searching for existing ones. Additionally, expanding the framework beyond furniture to more complex categories like vehicles or organic shapes remains a key objective.
Summary updated