Neural 3D Strokes: Creating Stylized 3D Scenes with Vectorized 3D Strokes

Hao-Bin Duan, Miao Wang, Yan-Xun Li, Yong-Liang YangCVPR 2024 · DOI · arXiv

Introduction

Neural 3D Strokes introduces a novel framework for representing and stylizing 3D scenes using vectorized primitives and splines. Unlike traditional methods that rely on voxels or point clouds, this approach utilizes a differentiable renderer and a specialized "error field" to place strokes artistically. The result is a 3D scene that exhibits significant geometric stylization while maintaining perfect multi-view consistency—a feat that remains challenging for standard Neural Radiance Fields (NeRFs).

The Challenge of 3D Stylization

While NeRFs have achieved remarkable success in photorealistic reconstruction, stylizing them often results in a "texture-on-geometry" look. Most existing style transfer methods merely repaint the colors of a scene, leaving the underlying density field untouched. This lacks the structural abstraction found in human art, such as thick brushstrokes or cubist forms. Furthermore, applying 2D stroke-based techniques to 3D renders often leads to "texture flickering," where strokes shift inconsistently as the camera moves. Neural 3D Strokes addresses these issues by defining the style within the 3D geometry itself.

Methodology: The Stroke Field

The core innovation is the Stroke Field, a 3D representation composed of $N$ parameterized vectorized strokes. This method moves away from discrete grids toward a mathematical definition of form.

1. 3D Stroke Palette

Strokes are defined using basic geometric primitives like spheres, boxes, and ellipsoids, as well as polynomial splines (Bezier and Catmull-Rom). These are represented as Signed Distance Fields (SDFs), providing a clean mathematical basis for the scene's structure.

2. Differentiable Rendering and Adaptive Smoothness

To optimize these strokes via gradient descent, the authors convert the discrete SDF boundaries into a smooth density field using a Laplace Cumulative Distribution Function (CDF). This creates a "fuzzy" boundary that allows gradients to flow effectively. Additionally, the system employs Adaptive Smoothness, adjusting stroke sharpness based on the camera ray's cone size to prevent aliasing and ensure visual clarity at varying distances.

3. The Error Field and Training Strategy

Optimization is guided by an "Error Field" that identifies regions where the current stroke-based reconstruction deviates from the target images. New strokes are automatically seeded in these high-error areas, solving the vanishing gradient problem that often plagues stroke-based optimization when primitives are far from their targets.

Key Insights: Geometry as Style

A fundamental insight of this work is that geometry is style. By restricting the "palette" to specific primitives, the scene naturally adopts a specific aesthetic. For example, using only axis-aligned boxes creates a voxel-art or LEGO-like appearance that traditional NeRFs cannot easily replicate.

Comparison between high-fidelity reconstructions and stylized versions using geometric primitives like boxes and ellipsoids.

Furthermore, because the scene is vectorized, it is resolution-independent. It can be rendered at any scale without losing the crispness of the underlying mathematical primitives.

Results and Applications

The method was evaluated on the Blender Synthetic and LLFF datasets. Among the stroke types, ellipsoids and cubic Bezier curves provided the best balance of fidelity and artistic flexibility. On synthetic scenes, ellipsoids achieved a PSNR of 21.78.

Beyond simple reconstruction, the framework enables advanced applications like Color Transfer and Text-driven Drawing. By integrating with CLIP, the system can generate 3D stroke-based objects from text prompts, such as a "sandwich" or "tomato," while maintaining the unique aesthetic of the chosen 3D primitives.

A painterly 3D scene of flowers demonstrating the artistic stylization achieved through vectorized 3D strokes.

Limitations and Future Work

While powerful, the method currently requires manual selection of stroke primitives to achieve a specific "look." Future research may involve learning these stroke shapes directly from artistic datasets to automate style selection. Additionally, representing highly complex scenes requires a large number of strokes, which increases computational overhead. The authors suggest that incorporating global loss functions, such as optimal transport, could improve stroke distribution and efficiency in future iterations.

Summary updated

BibTeX
@inproceedings{duan2024neural,
  author = {Hao-Bin Duan and Miao Wang and Yan-Xun Li and Yong-Liang Yang},
  title = {{Neural 3D Strokes: Creating Stylized 3D Scenes with Vectorized 3D Strokes}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2024},
  doi = {10.1109/cvpr52733.2024.00501},
}
Figure