Abstract
Recent advancements in leveraging pre-trained 2D diffusion models achieve the generation of high-quality novel views from a single in-the-wild image. However, existing works face challenges in producing controllable novel views due to the lack of information from multiple views. In this paper, we present DreamComposer++, a flexible and scalable framework designed to improve current view-aware diffusion models by incorporating multi-view conditions. Specifically, DreamComposer++ utilizes a view-aware 3D lifting module to extract 3D representations of an object from various views. These representations are then aggregated and rendered into the latent features of target view through the multi-view feature fusion module. Finally, the obtained features of target view are integrated into pre-trained image or video diffusion models for novel view synthesis. Experimental results demonstrate that DreamComposer++ seamlessly integrates with cutting-edge view-aware diffusion models and enhances their abilities to generate controllable novel views from multi-view conditions. This advancement facilitates controllable 3D object reconstruction and enables a wide range of applications.
Introduction
Generating high-fidelity 3D content from 2D images has long been a "guessing game" for artificial intelligence. While modern diffusion models excel at hallucinating missing details, single-view inputs often lead to structural inconsistencies, such as the "Janus problem," where a model generates multiple faces or mismatched textures on a single object. Existing view-aware models, like Zero-1-to-3, are primarily optimized for single-view inputs and lack a native mechanism to reconcile information from multiple perspectives.
DreamComposer++ addresses this limitation by providing a flexible, "plug-and-play" framework that upgrades existing 2D and video diffusion models to handle an arbitrary number of input views. By lifting 2D images into a shared 3D feature space and fusing them through a target-aware rendering process, the framework ensures global consistency and provides users with precise control over 3D reconstruction.
Methodology
The DreamComposer++ framework operates through three primary stages designed to integrate seamlessly with existing architectures like Stable Diffusion or SV3D:
1. Target-Aware 3D Lifting
The process begins by taking 2D latent features from multiple input images and "lifting" them into a 3D tri-plane representation. Tri-planes organize data into three perpendicular planes (XY, XZ, and YZ), offering an efficient way to store spatial information. Crucially, this lifting is "target-aware," focusing specifically on the features most relevant to the novel viewpoint the user intends to generate.
2. Multi-View Feature Fusion
To resolve conflicts between overlapping or differing input views, the framework employs a composited volume rendering approach. The model samples points along a ray originating from the target camera and aggregates features from all available input views. An adaptive weighting strategy is used here: input views with camera angles closer to the target view are given higher priority, ensuring that the most relevant visual data informs the final output.
3. Target-View Feature Injection
Once the fused 3D-aware features are obtained, they must be integrated into the diffusion model's UNet. Rather than using standard residual injection (simply adding features together), DreamComposer++ utilizes a "Concat-conv" approach. By concatenating the new features with the UNet's internal state and passing them through a learnable convolution layer, the model achieves richer spatial interaction and superior control over fine details. For video-based models like SV3D, the framework incorporates View-aware Attention and Shifted Window Cross Attention to maintain temporal consistency and prevent jittering across frames.
Experimental Results
The researchers evaluated DreamComposer++ by integrating it into three prominent models: Zero-1-to-3, SyncDreamer, and SV3D. Training was conducted on the Objaverse dataset (800,000 3D objects), with evaluation performed on Google Scanned Objects (GSO) and CO3Dv2.
Quantitative and Qualitative Improvements
- Reconstruction Quality: On the GSO dataset, adding DreamComposer++ to Zero-1-to-3 improved the Peak Signal-to-Noise Ratio (PSNR) from 20.82 to 25.25, a significant jump in reconstruction fidelity.
- Interactive Controllability: The framework enables a "correction-based" workflow. If a user generates a back view they find unsatisfactory, they can provide a single "correction" image for that angle. DreamComposer++ then updates the entire 3D representation to remain consistent with both the original front view and the new back view.
- Character Modeling: The model demonstrated high efficacy in converting 2D character "turnaround" sketches into consistent 3D models, maintaining texture fidelity that surpasses previous single-view or naive multi-view methods.
Key Insights
- Scalability: Performance scales linearly with the number of input views. The framework can handle anywhere from two to six or more views, with each additional view increasing the accuracy and controllability of the output.
- The Concat-conv Advantage: Analysis showed that simple feature addition is insufficient for complex 3D tasks. The concatenation and convolution method allows the model to better adjust both spatial and channel-wise information.
- The Lifting Bottleneck: Through a "bottleneck analysis," the authors discovered that replacing generated 3D features with ground-truth features led to a massive jump in quality. This suggests that the primary limit on current 3D generation is not the diffusion model itself, but the ability of the lifting module to create perfect 3D representations from 2D pixels.
Limitations and Future Work
While DreamComposer++ represents a significant step forward, it faces challenges regarding computational overhead due to the volume rendering required for feature fusion. Additionally, the model remains dependent on the underlying priors of the base diffusion model; if the base model has no experience with a specific object category, the framework's ability to correct geometry is limited. Future work will focus on refining the 3D lifting module to further close the gap between generated and ground-truth features.
Summary updated