Abstract
Estimating the 3D pose of objects from 2D images is a fundamental task in computer vision, yet it is hindered by the high cost of obtaining precise 3D annotations. This paper introduces a novel framework to overcome this bottleneck. First, P3D-Diffusion leverages generative diffusion models to "paint" photorealistic textures onto 3D-consistent graphics renders, creating a high-fidelity synthetic dataset. Second, CC3D (Cross-domain 3D Consistency) provides an unsupervised domain adaptation mechanism that aligns features between synthetic and real domains in 3D space. Our approach matches state-of-the-art performance using only 10% of the typical real-world training data and exhibits significant robustness to out-of-distribution (OOD) scenarios.
Introduction
Category-level 3D pose estimation aims to determine the 3D orientation of objects within a specific category (e.g., cars, chairs) from a single 2D image. While deep learning has advanced this field, the reliance on large-scale, 3D-annotated datasets remains a major hurdle. Manual annotation is labor-intensive and often inaccurate. Synthetic data offers a potential solution, but the "domain gap"—the visual difference between clean computer graphics and complex real-world scenes—often leads to poor generalization. Furthermore, models often overfit to "spurious correlations," such as associating cars exclusively with asphalt roads.
P3D-Diffusion: High-Fidelity Synthetic Data Generation
To bridge the domain gap, the authors propose P3D-Diffusion, a pipeline that transforms simple 3D renders into photorealistic images while preserving geometric ground truth.
The process involves three key stages:
- Graphics-Based Rendering: 3D CAD models are rendered to produce basic images with precise 3D pose annotations.
- Graphics-Guided Style Transfer: Canny edge maps are extracted from the renders and used as structural guides for a pre-trained diffusion model (e.g., Stable Diffusion with ControlNet). This ensures the generated image matches the 3D geometry exactly.
- OOD-Aware Generation: To prevent the model from learning environmental biases, objects are placed in diverse, non-standard backgrounds (e.g., a car in a forest or a boat on a road). This forces the model to focus on the object's shape rather than its context.
CC3D: Cross-Domain 3D Consistency
Even with high-quality synthetic data, some domain gap persists. The authors introduce CC3D, an unsupervised domain adaptation method built upon Neural Mesh Models (NMMs). NMMs represent objects as 3D cuboid meshes where each vertex holds a learnable feature.
CC3D employs a domain-consistency loss that encourages the features extracted from real images to align with those learned from synthetic images at the same 3D mesh locations. Additionally, a self-consistency mechanism uses high-confidence predictions on unlabeled real data as "pseudo-labels" to further refine the model. By aligning features in 3D space rather than 2D pixels, the model achieves a more robust geometric understanding.
Experimental Results
The proposed framework was evaluated on the PASCAL3D+ benchmark and the OOD-CV dataset for robustness.
Data Efficiency and Performance
A standout result is the framework's data efficiency. As shown in the table below, the P3D-Diffusion + CC3D combination significantly outperforms previous baselines in few-shot settings.
Using only a small number of real-world training images (e.g., 7, 20, or 50 annotations per category), the model achieves a mean accuracy ($ACC_{\frac{\pi}{6}}$) of 81.1%, nearly doubling the performance of the baseline NeMo (53.1%) under similar constraints. When provided with 50% of the full dataset, it surpasses existing state-of-the-art methods by a substantial margin.
Robustness to OOD Scenarios
On the OOD-CV benchmark, which tests models against nuisance factors like extreme weather and unusual contexts, the method demonstrated remarkable resilience. By training on OOD-aware synthetic data, the model avoids spurious correlations, maintaining high accuracy where traditional models fail. For example, in the "Weather" and "Context" categories, the proposed method maintained much higher accuracy than standard baselines.
Conclusion
This work demonstrates that generative AI can serve as a powerful tool for geometric computer vision tasks. By combining diffusion-based data augmentation with 3D-aware domain adaptation, the authors significantly reduce the need for expensive real-world annotations. Future research may focus on expanding these techniques to organic or "long-tail" object categories where CAD models are less readily available, potentially moving toward a future where real-world 3D labels are no longer a requirement for robust pose estimation.
Summary updated