Abstract
StyleMesh is a novel framework designed to apply artistic style transfer to 3D reconstructions of entire indoor scenes. Unlike traditional 2D style transfer methods that suffer from temporal flickering and spatial inconsistencies when applied to 3D environments, StyleMesh incorporates depth and surface normal information directly into the optimization process. By optimizing an explicit RGB texture map using a Laplacian Pyramid representation, the method ensures that stylization patterns remain consistent in scale and orientation across different viewpoints. Our approach produces high-quality, spatially stable stylized textures that can be rendered in real-time in standard graphics engines.
Introduction
Neural Style Transfer (NST) has revolutionized the way we manipulate 2D images, yet extending these techniques to 3D environments remains a significant challenge. Applying 2D style transfer frame-by-frame results in visual "flickering" due to lack of temporal coherence. Furthermore, optimizing a 3D texture using standard 2D losses introduces perspective distortion—where patterns stretch at grazing angles—and scale inconsistency—where pattern sizes vary based on camera distance.
StyleMesh addresses these issues by creating a single, high-quality, explicit RGB texture for a 3D mesh. The goal is a stylization that appears consistent from any viewpoint, effectively "locking" the artistic style to the physical geometry of the scene.
Methodology
StyleMesh utilizes an optimization-based approach that bridges 2D style losses with 3D geometry. Instead of a feed-forward network, the system optimizes the texels of a Laplacian Pyramid texture directly through differentiable rendering.
Technical Innovations
- Depth-Aware Render Pyramid: To maintain consistent pattern sizes in world space, the scene is rendered at multiple resolutions. Depth maps segment the image: foreground pixels are compared against high-resolution style features, while background pixels use low-resolution ones. This ensures artistic elements maintain a constant physical size regardless of camera distance.
- Angle-Aware Filtering: To mitigate stretching artifacts, the system calculates the angle between surface normals and the view direction. Fine-grained style details are only applied to surfaces viewed from near-orthogonal angles. Grazing angles receive only coarse stylization, which is refined as the camera moves to a better vantage point.
- Differentiable Rendering and Gradient Scaling: The pipeline uses differentiable rendering to backpropagate 2D image-space losses into the 3D texture. Trilinear Mipmap interpolation and continuous weighting are employed to scale gradients, preventing harsh seams between different depth or angle zones.
Evaluation and Results
The framework was evaluated on the ScanNet and Matterport3D datasets, which provide diverse RGB-D scans of indoor environments.
Key Insights
- Resolution as a Proxy for Scale: By varying resolution based on depth, StyleMesh effectively maps the fixed receptive field of a CNN to a physical scale in the 3D world.
- Geometry as a Regularizer: Using surface normals to gate style loss prevents the optimization from being corrupted by distorted viewpoints.
- Explicit Representation: Because StyleMesh produces a standard texture map, the final stylized scene can be rendered in real-time using traditional graphics pipelines like Unity or Unreal Engine after the initial optimization (approx. 3 hours on an RTX 3090).
Quantitative and Qualitative Performance
Quantitatively, StyleMesh achieved a significantly lower L1 reprojection error (1.225) compared to baselines like NMR (1.565) and DIP (1.396), demonstrating superior spatial stability. In a user study, participants overwhelmingly preferred StyleMesh, noting better maintenance of scale (~75% preference) and fewer visible stretches (~80% preference) compared to existing methods.
Conclusion and Future Work
While StyleMesh provides a robust solution for 3D scene stylization, it is currently limited by its per-scene optimization time and dependence on the quality of the initial 3D reconstruction. Future work may explore generative models to accelerate the stylization process or integrate mesh completion techniques to handle imperfect scans. Despite these limitations, StyleMesh represents a significant step toward immersive, artistically stylized 3D environments.
Summary updated