Abstract: We propose a novel technique for producing high-quality 3D models that match a given target object image or scan. Our method is based on retrieving an existing shape from a database of 3D models and then deforming its parts to match the target shape. Unlike previous approaches that independently focus on either shape retrieval or deformation, we propose a joint learning procedure that simultaneously trains the neural deformation module along with the embedding space used by the retrieval module. This enables our network to learn a deformation-aware embedding space, so that retrieved models are more amenable to match the target after an appropriate deformation. In fact, we use the embedding space to guide the shape pairs used to train the deformation module, so that it invests its capacity in learning deformations between meaningful shape pairs. Furthermore, our novel part-aware deformation module can work with inconsistent and diverse part-structures on the source shapes. We demonstrate the benefits of our joint training not only on our novel framework, but also on other state-of-the-art neural deformation modules proposed in recent years. Lastly, we also show that our jointly-trained method outperforms various non-joint baselines.
Introduction
Creating high-quality 3D models from a single image or a messy point cloud is a long-standing challenge in computer vision and graphics. While generative AI can synthesize new shapes, they often lack the clean topology and precise details of artist-created meshes. To address this, researchers frequently employ a "retrieve-and-deform" strategy: finding a similar model in a database and adjusting its parts to fit the target.
However, these two steps are typically treated as independent silos. A retrieval system might select a model that looks visually similar in its static state but is topologically difficult to deform into the target. Conversely, a deformation module trained on random pairs may never specialize in the specific nuances required to align the candidates a retrieval system actually selects. This research argues that retrieval and deformation should be treated as "equal citizens" in a joint optimization problem.
Methodology
The core innovation is a joint learning procedure where the retrieval embedding space and the neural deformation module are trained simultaneously. This allows the modules to "talk" to each other during the learning process, leading to several key advancements:
- Deformation-Aware Retrieval: The system learns a latent embedding space where the distance between a target and a source model is based on how well the source can fit the target after deformation, rather than just visual similarity.
- Retrieval-Aware Deformation: The deformation module is trained specifically on the pairs of shapes that the retrieval module is likely to pick, allowing the network to specialize its capacity on meaningful matches.
- Soft Retrieval for Training: To make the pipeline differentiable, the authors use "soft retrieval," sampling multiple candidates from a probability distribution in the embedding space to allow gradient flow.
- Structure-Aware Deformation Module: This module handles heterogeneous shapes with inconsistent part structures. It uses a part-based MLP to predict translation and scaling while enforcing connectivity constraints to ensure structural integrity.
Results and Evaluation
The framework was evaluated on major furniture categories (chairs, tables, and cabinets) using datasets such as ShapeNet, PartNet, and Scan2CAD. The joint approach consistently outperformed non-joint baselines across both image-to-mesh and point-cloud-to-mesh tasks.
Quantitatively, on the image-to-mesh benchmark, the joint method achieved a Chamfer distance of 1.005 for chairs, compared to 1.926 for vanilla retrieval and 1.216 for a non-joint deformation-aware baseline. Qualitatively, visual results demonstrate much tighter alignment to target silhouettes. For instance, when reconstructing a chair from an image, the joint method correctly identifies and scales the specific height of the backrest and the spread of the legs, whereas baselines often produce floating parts or poor global proportions. The system also proved effective at completing noisy, partial real-world scans from the Scan2CAD dataset.
Discussion and Future Work
A primary insight of this work is that joint training discovers "hidden" matches—database shapes that might look wrong globally but possess local details that match the target perfectly after deformation. By biasing deformation training toward retrieved pairs, the model focuses entirely on subtle tweaks needed for similar objects rather than wasting capacity on irrelevant transformations.
While powerful, the method currently relies on Chamfer distance for geometric fit, which can sometimes prioritize geometric alignment over semantic correctness. Future work may incorporate manual part annotations to improve semantic similarity and explore advanced graph-based neural architectures to better capture complex relationships between object parts.
Summary updated