TL;DR
DreamComposer++ is a flexible, "plug-and-play" framework that upgrades existing 2D and video diffusion models to handle multiple input views for 3D generation. By lifting 2D images into a shared 3D feature space and fusing them, it eliminates common artifacts like the "Janus problem" (multiple faces) and provides users with precise, multi-view control over 3D object reconstruction.
What Problem Does This Paper Address?
Generating a 3D object from a single image is a bit like a guessing game. If a model only sees the front of a car, it has to "hallucinate" what the back looks like. While modern diffusion models are great at this, they often produce inconsistent results—like a character having two faces or textures that don't match up when you rotate the camera.
Existing "view-aware" models (like Zero-1-to-3) are mostly designed for single-view inputs. When you try to give them more information (e.g., a front and a side view), they don't have a native way to combine that data into a coherent 3D signal. DreamComposer++ tackles this by creating a scalable way to inject an arbitrary number of viewpoint conditions into pre-trained models.
Methods
The framework consists of three main stages that can be integrated into existing architectures:
- Target-Aware 3D Lifting: The model takes 2D latent features from multiple input images and "lifts" them into 3D tri-planes. This is an efficient 3D representation that organizes data into three perpendicular planes (XY, XZ, YZ). Crucially, this lifting is "target-aware," meaning it focuses on the features most relevant to the specific novel view the user wants to generate.
- Multi-View Feature Fusion: Since different input views might conflict or overlap, the model uses a composited volume rendering approach. It samples points along a ray from the target camera and aggregates features from all input views. It uses an adaptive weighting strategy: input views that are closer in angle to the target view are given more "say" in the final feature.
- Target-View Feature Injection: These fused 3D-aware features are injected into the UNet of a diffusion model (like Stable Diffusion or SV3D). Unlike standard methods that just add features together (residual injection), the authors use a "Concat-conv" approach. They concatenate the new features with the UNet's internal state and pass them through a learnable convolution layer, allowing for much richer interaction and better spatial control.
For video models (like SV3D), they added View-aware Attention and Shifted Window Cross Attention to ensure that as the camera moves in a video, the object remains temporally consistent and doesn't "jitter" or warp.
Datasets
- Training: The model was trained on Objaverse, a massive dataset containing roughly 800,000 3D objects.
- Evaluation: To test how well the model generalizes to "in-the-wild" or unseen data, they used:
- Google Scanned Objects (GSO): High-quality scans of common household items.
- CO3Dv2 (Common Objects in 3D): A dataset of real-world category-specific 3D reconstructions.
Key Insights and Ideas
- Scalability is Key: The architecture is designed to handle any number of input views (from 2 to 6+). The authors found that performance scales linearly: the more views you provide, the higher the accuracy and controllability.
- The "Concat-conv" Advantage: The paper demonstrates that simply adding features (residual) isn't enough for complex 3D tasks. By concatenating and then convolving, the model can better adjust both spatial and channel-wise information, leading to sharper details.
- Lifting is the Bottleneck: In a "Bottle Neck Analysis," the authors replaced their generated 3D features with ground-truth features. The quality jumped significantly, suggesting that the primary limit on current 3D generation isn't the diffusion model itself, but the "lifting" module's ability to create perfect 3D representations from 2D pixels.
Key Results
The researchers "plugged" DreamComposer++ into three major models: Zero-1-to-3, SyncDreamer, and SV3D.
- Quantitative Boost: On the GSO dataset, adding DreamComposer++ to Zero-1-to-3 improved the PSNR (a measure of image reconstruction quality) from 20.82 to 25.25.
- Controllability: The model enables "interactive" 3D generation. A user can provide a front view, see a generated back view they don't like, provide a "correction" image for the back, and the model will seamlessly update the entire 3D object to match both.
- Character Modeling: The framework proved highly effective at turning 2D character "turnaround" sketches (common in animation) into consistent 3D models with much higher texture fidelity than previous methods.
Limitations and Future Work
- The Lifting Module: As noted in their bottleneck analysis, the 3D lifting module is the weakest link. Improving how 2D features are projected into 3D space is a primary goal for future versions.
- Computational Cost: While tri-planes are efficient, processing multiple views and performing volume rendering for feature fusion adds computational overhead compared to single-view models.
- Dependency on Priors: The model still relies heavily on the base diffusion model's learned priors. If the base model hasn't seen a specific type of object before, DreamComposer++ can only do so much to fix the geometry.
Summary updated