Locally Stylized Neural Radiance Fields

Hong-Wing Pang, Binh-Son Hua, Sai-Kit Yeung

Abstract: In recent years, there has been increasing interest in applying stylization on 3D scenes from a reference style image, in particular onto neural radiance fields (NeRF). While performing stylization directly on NeRF guarantees appearance consistency over arbitrary novel views, it is a challenging problem to guide the transfer of patterns from the style image onto different parts of the NeRF scene. In this work, we propose a stylization framework for NeRF based on local style transfer. In particular, we use a hash-grid encoding to learn the embedding of the appearance and geometry components, and show that the mapping defined by the hash table allows us to control the stylization to a certain extent. Stylization is then achieved by optimizing the appearance branch while keeping the geometry branch fixed. To support local style transfer, we propose a new loss function that utilizes a segmentation network and bipartite matching to establish region correspondences between the style image and the content images obtained from volume rendering. Our experiments show that our method yields plausible stylization results with novel view synthesis while having flexible controllability via manipulating and customizing the region correspondences.

Introduction

3D style transfer aims to "paint" a 3D scene so it resembles the style of a reference image (like a Van Gogh painting) while remaining consistent as the camera moves. While Neural Radiance Fields (NeRF) are excellent at maintaining this multi-view consistency, existing stylization methods often struggle with locality. Most current methods apply a "global" style, which frequently results in the same repetitive texture being plastered over the entire scene—for example, the floor, walls, and furniture might all end up with the exact same brushstroke pattern.

Comparison of local stylization (Ours) vs. global stylization (ARF). The lines indicate the region correspondences established between the style image and the 3D scene.

This paper tackles the challenge of establishing meaningful correspondences between local regions of a 3D scene and local patterns in a style image. By combining unsupervised segmentation with bipartite matching, the authors achieve 3D stylization that is more diverse, controllable, and visually consistent than previous global approaches.

Methodology

The authors propose a two-stage pipeline (Reconstruction and Stylization) built on a dual-branch NeRF architecture. The model splits the NeRF into a geometry branch (predicting density) and an appearance branch (predicting color). During stylization, the geometry branch is frozen to ensure the 3D structure of the scene remains unchanged while only the "paint" is updated.

To support local style transfer, the system utilizes several key components:

  • Hash-Grid Encoding: A multi-resolution hash-grid (similar to Instant-NGP) represents the scene, speeding up training and allowing multiple styles to be stored within a single parametric embedding.
  • Unsupervised Segmentation: The style image is segmented using the Segment Anything Model (SAM), while the 3D scene is segmented during reconstruction using an unsupervised clustering method to ensure consistency across views.
  • Bipartite Matching: To decide which style region goes with which scene region, the authors formulate a matching problem using the Hungarian algorithm to find an optimal one-to-one mapping based on feature similarity and spatial position.
  • Region-Aware Style Loss: Instead of a global loss, the style loss is calculated only between matched regions, ensuring that specific scene elements (like trees) receive appropriate textures from the painting (like leaves).

Key Results and Insights

The method was evaluated on standard benchmarks including LLFF (Local Light Field Fusion) and the Replica Dataset. Qualitative results show that the method successfully avoids the "blanketing" effect seen in baselines like ARF (Artistic Radiance Fields), applying distinct textures to different objects within the same scene.

Stylization results on an indoor scene using Van Gogh's 'Starry Night' as a reference style.

Key insights from the study include:

  • Assignment over Search: By using bipartite matching instead of Nearest-Neighbor Feature Matching (NNFM), the model avoids "mode collapse" where the entire scene adopts only the most dominant style feature.
  • Controllability: Users can manually override automatic assignments, allowing for custom "re-wiring" of correspondences between the painting and the 3D scene.
  • Efficiency: Leveraging hash-grid encodings reduces the stylization process to a matter of minutes, a significant improvement over older MLP-based NeRF stylization methods.

In a user study with 23 participants, the proposed method was preferred over ARF 72.3% of the time on real-world scenes and 64.1% on synthetic scenes.

Limitations and Future Work

While much faster than previous work, the optimization still takes several minutes per style, making it unsuitable for live, interactive stylization. Additionally, the current implementation relies on the bounded volume logic of Instant-NGP, which presents challenges for applying the method to "unbounded" outdoor scenes with infinite backgrounds. Future research may also explore "zero-shot" versions that can apply any style at inference time without requiring optimization for each new style image.

Summary updated

Abstract

In recent years, there has been increasing interest in applying stylization on 3D scenes from a reference style image, in particular onto neural radiance fields (NeRF). While performing stylization directly on NeRF guarantees appearance consistency over arbitrary novel views, it is a challenging problem to guide the transfer of patterns from the style image onto different parts of the NeRF scene. In this work, we propose a stylization framework for NeRF based on local style transfer. In particular, we use a hash-grid encoding to learn the embedding of the appearance and geometry components, and show that the mapping defined by the hash table allows us to control the stylization to a certain extent. Stylization is then achieved by optimizing the appearance branch while keeping the geometry branch fixed. To support local style transfer, we propose a new loss function that utilizes a segmentation network and bipartite matching to establish region correspondences between the style image and the content images obtained from volume rendering. Our experiments show that our method yields plausible stylization results with novel view synthesis while having flexible controllability via manipulating and customizing the region correspondences.
BibTeX
@inproceedings{pang2023locally,
  author = {Hong-Wing Pang and Binh-Son Hua and Sai-Kit Yeung},
  title = {{Locally Stylized Neural Radiance Fields}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2023},
  url = {https://openreview.net/forum?id=gV2mo6Tk5m},
}
Figure