Abstract
Neural Radiance Fields (NeRFs) have transformed 3D scene representation, yet applying artistic styles to these environments remains computationally expensive and often lacks fine-grained control. We present ConRF, a novel framework for zero-shot artistic stylization of 3D scenes. By mapping CLIP’s multi-modal embeddings into a refined VGG-based style space, ConRF enables instantaneous transformation of entire scenes or specific objects using text prompts or reference images. Our approach eliminates the need for per-style retraining while maintaining high visual fidelity and multi-view consistency.
Introduction
While NeRFs excel at photorealistic reconstruction, "painting" these scenes in diverse artistic styles is challenging. Existing methods typically face two hurdles: the "retraining bottleneck," where new styles require hours of optimization, and limited stylistic control, often restricted to basic color shifts rather than complex textural transfers. ConRF addresses these by leveraging the semantic power of CLIP (Contrastive Language-Image Pre-training) while overcoming its inherent ambiguity regarding fine-grained textures.
Methodology
The ConRF architecture introduces several key innovations to bridge the gap between high-level semantics and low-level artistic texture:
CLIP-to-VGG Mapping
Raw CLIP features are effective for semantic matching but often lack the textural detail necessary for artistic style transfer. ConRF introduces a mapping network that transforms CLIP embeddings into the style space of a pre-trained VGG network. By focusing on the mean and standard deviation of VGG features, the model captures the "feel" of a style—its texture and color distribution—while filtering out semantic noise.
Local Stylization and 3D Selection
To allow for targeted editing, ConRF utilizes a 3D volumetric representation. A 3D selection volume is generated by calculating the similarity between 3D points and a content text prompt (e.g., "the flower"). This creates a mask, enabling the application of styles to specific regions rather than the entire scene.
Multi-spatial Feature Extraction
To improve pixel-level accuracy, which is a known weakness of global CLIP features, ConRF employs a sliding-window approach during training. By extracting features from patches of varying sizes, the model better understands the correspondence between specific scene parts and text descriptions.
Experimental Results
ConRF was evaluated on standard benchmarks including LLFF and Synthetic NeRF.
Quantitative Performance
In multi-view consistency tests, ConRF significantly outperformed the baseline StyleRF. It achieved a 31% improvement in short-range LPIPS and a 59.4% improvement in long-range LPIPS, indicating superior stability as the camera moves through the stylized environment.
Qualitative Comparison
Unlike previous methods like CLIP-NeRF, which often struggle to move beyond simple color modifications, ConRF successfully transfers complex textures and brushstrokes. User studies involving 25 participants confirmed that ConRF produces results more faithful to prompts and more visually consistent than other state-of-the-art methods like Ref-NPR or SNeRF.
Conclusion and Future Work
ConRF demonstrates a powerful approach to zero-shot 3D stylization, offering a versatile tool for artists and developers. While highly effective, the model remains dependent on CLIP's internal representations and currently performs best in "face-forwarding" scenes. Future research will explore integrating generative capabilities to allow the model to synthesize entirely new content beyond simple style transfer.
Summary updated