Introduction
3D Gaussian Splatting (3DGS) has revolutionized real-time novel-view synthesis, yet "editing" these scenes remains a significant challenge. Most existing stylization techniques are essentially "skin-deep"—they can change the texture or color of an object but struggle to alter its fundamental geometry. Morpheus is a new framework that allows users to stylize 3D scenes using text prompts, going beyond simple color changes to actually modify the underlying shape while maintaining multi-view consistency.
Methodology: Joint Color and Depth Stylization
Morpheus tackles the challenge of geometric editing by treating stylization as a joint color and depth problem. The pipeline consists of three main stages:
- The RGBD Diffusion Model: The authors modified Stable Diffusion 2.1 to generate both RGB images and corresponding depth maps (RGBD). Crucially, they introduced separate "noise schedules" for color and depth. This provides a "knob" for geometric creativity, allowing users to choose whether to change colors significantly while keeping the original shape or to push depth stylization to reinvent the geometry.
- Autoregressive Stylization with Warp ControlNet: To maintain multi-view consistency, Morpheus stylizes a sequence of views one by one. For each new frame, it "warps" previous stylized frames into the current camera view using depth information. A specialized Warp ControlNet then guides the diffusion model to fill in gaps while staying true to the established style.
- Depth-Informed Feature Sharing: To prevent artifacts like "flickering" or duplicated features, the model uses 4D heatmaps based on the scene's geometry to share features across frames, ensuring stylized features stay anchored to the correct physical location in 3D space.
Results and Comparison
Morpheus outperforms state-of-the-art baselines like GaussCtrl and Instruct-GS2GS across both quantitative and qualitative metrics. It achieved a CLIP Direction Similarity of 0.175, significantly beating GaussCtrl (0.123), indicating better adherence to text prompts. In terms of consistency, it scored 0.606, outperforming all baselines.
As illustrated in the comparison, Morpheus can dramatically alter the scene's geometry—such as adding the rounded structures of a Hobbit village or the jagged icicles of an ice castle—whereas baseline methods often struggle to move beyond texture-level changes. A user study with 31 participants consistently preferred Morpheus for its aesthetic quality and its ability to follow complex style prompts.
Limitations and Future Work
While Morpheus represents a major step forward, it remains dependent on the chosen camera trajectory during stylization; areas not covered by the path may remain unstylized. Additionally, errors in the initial 3DGS reconstruction can be amplified during the stylization process. Future research will explore faster diffusion architectures, such as Latent Consistency Models, to move toward near-interactive 3D stylization speeds.
Summary updated