Abstract
Neural Radiance Field (NeRF) is a powerful tool to faithfully generate novel views for scenes with only sparse captured images. Despite its strong capability for representing 3D scenes and their appearance, its editing ability is very limited. In this paper, we propose a simple but effective extension of vanilla NeRF, named PaletteNeRF, to enable efficient color editing on NeRF-represented scenes. Motivated by recent palette-based image decomposition works, we approximate each pixel color as a sum of palette colors modulated by additive weights. Instead of predicting pixel colors as in vanilla NeRFs, our method predicts additive weights. The underlying NeRF backbone could also be replaced with more recent NeRF models such as KiloNeRF to achieve real-time editing. Experimental results demonstrate that our method achieves efficient, view-consistent, and artifact-free color editing on a wide range of NeRF-represented scenes.
Introduction
Neural Radiance Fields (NeRFs) have revolutionized how we capture and render 3D scenes, but they remain notoriously difficult to edit. Because a NeRF "bakes" geometry, lighting, and color into the weights of a neural network, changing the color of a single object usually requires expensive retraining or the use of text-to-image prompts that often introduce noise or unintended changes to the background.
PaletteNeRF offers a clever way out. By combining traditional palette-based image decomposition with the volumetric power of NeRFs, the authors enable real-time, view-consistent color editing without ever needing to touch the underlying neural network after it is trained. It decomposes a 3D scene’s appearance into a small set of palette colors and learned mixing weights, allowing users to recolor complex scenes instantly and realistically just by swapping colors in a palette.
Methods
The core innovation of PaletteNeRF is shifting the network's objective. Instead of predicting a 3D RGB vector for every point in space, the MLP predicts a $K$-dimensional vector of additive weights ($w$).
- Palette Extraction: Before training, the system analyzes the input images to extract a global palette ($V$) of $K$ colors and computes per-pixel mixing weights using Mean Value Coordinates (MVC).
- Modified MLP: The authors modify the standard NeRF architecture, changing the final layer from an RGB output to a $K$-dimensional weight output.
- Two-Stage Training: To ensure high accuracy, they use a "pretrain + finetune" strategy. They first train a vanilla NeRF to learn the scene's geometry and color. Then, they freeze the geometry (density) layers and finetune only the color-related layers to predict the MVC weights.
- Rendering & Editing: To render a view, the model outputs weights which are then multiplied by the palette colors ($I = V \cdot W$). To edit, the user simply changes the values in the palette $V$. Since the weights $W$ are already learned and view-consistent, the new colors "stick" to the 3D geometry perfectly.
Key Results and Insights
PaletteNeRF achieves reconstruction quality nearly identical to vanilla NeRF. For the Lego scene, it maintained a PSNR of ~30.1 dB compared to the original's ~31.1 dB, while gaining full editability. Unlike previous methods that picked the "closest" palette color (causing harsh edges), PaletteNeRF uses continuous weights, preserving the smooth gradients and subtle shading essential for realism.
Because the method only changes the final output layer, it is backbone-agnostic. It can be plugged into faster NeRF variants like KiloNeRF, enabling real-time editing at interactive frame rates with only a 4% performance overhead. Furthermore, for synthetic scenes, the authors introduced second-order weights to account for indirect lighting, allowing "color bleeding" effects to update automatically when a surface color is changed.
Conclusion
PaletteNeRF provides an intuitive and efficient framework for color editing in 3D scenes. By decoupling color values from their spatial proportions, it enables artists to recolor complex objects with the simplicity of a color picker. While currently limited to global edits and static geometry, it represents a significant step toward making NeRFs a practical tool for digital content creation.
Summary updated