Abstract
Traditional neural style transfer (NST) operates in the pixel domain, which often fails to capture the structural essence of physical painting. We propose a shift toward optimizing parameterized brushstrokes—specifically quadratic Bézier curves—using a novel differentiable renderer. This approach introduces a powerful inductive bias that mimics human artistic techniques, resulting in more "painterly" outputs. Furthermore, our method enables intuitive user control over stroke orientation through a projection loss. Experimental results demonstrate that our stroke-based representation outperforms pixel-based methods in both automated "deception" metrics and human preference studies.
Beyond Pixels: The Case for Brushstrokes
Since the introduction of Neural Style Transfer by Gatys et al., the field has largely treated stylization as a pixel-optimization problem. While effective at matching local textures, pixel-based methods are fundamentally "unnatural" for representing art. In the physical world, paintings are constructed from discrete, intentional brushstrokes, not independent color points.
Pixel-based optimization often struggles with structural coherence. Because pixels are independent, the model cannot easily "move" a visual element; it can only adjust intensities in place. By contrast, a stroke-based representation allows for spatial relocation—strokes can physically move, stretch, and rotate during optimization to better fit the target composition.
Methodology: Differentiable Stroke Rendering
The core of our approach is the move from a pixel grid to a set of parameterized brushstrokes. Each stroke is modeled as a quadratic Bézier curve defined by three control points (location), a color (RGB), and a width.
To integrate these strokes into a neural network optimization loop, we developed a lightweight differentiable renderer. Unlike previous methods that might use a heavy pre-trained network to simulate painting, our renderer uses explicit mathematical functions:
- Distance Calculation: It computes the distance from every pixel to the nearest stroke curve.
- Soft Masking: A sigmoid function creates a soft, differentiable mask for each stroke, allowing gradients to flow back to the control points.
- Layering: A softmax operation with high temperature determines stroke layering, effectively deciding which stroke appears "on top" at any given pixel.
The optimization process utilizes standard VGG-19 content and style losses. However, instead of updating pixel values, the gradients update the parameters of the Bézier curves. A final pixel-level refinement pass is used to blend strokes and add fine-grained textures.
Empowering the Artist: User-Guided Flow
One significant advantage of explicit stroke objects is the ability to introduce user agency. We implemented a "projection loss" that allows users to draw simple guide curves on the content image. This loss forces the optimized brushstrokes to align their tangents with the user's input, enabling custom "flow" in the final artwork—such as making a sky swirl or water ripple in a specific direction.
As shown in the figure above, increasing the number of strokes (from 2000 to 5000) increases the level of detail. More importantly, the bottom row demonstrates how user-defined guides (d) can drastically alter the orientation of strokes (e, f) to create a more dynamic and intentional composition compared to the unguided versions (b, c).
Quantitative and Qualitative Success
Our method was evaluated against state-of-the-art pixel-based and stroke-based models:
- Deception Rate: Using a pre-trained artist-classification network, our method achieved a score of 0.588, significantly higher than Gatys et al. (0.389).
- Human Evaluation: In a study where participants distinguished real artwork from generated crops, our "Human Deception Rate" was 26.8%, nearly doubling the performance of previous methods.
- Reconstruction Accuracy: When reconstructing target images, our renderer achieved a 49% lower Mean Squared Error (MSE) for 1000 strokes compared to previous "Learning to Paint" (LTP) models, proving the efficiency of our differentiable representation.
Conclusion
By rethinking style transfer as a stroke-parameterization problem rather than a pixel-shuffling one, we bridge the gap between digital optimization and traditional artistic practice. The resulting images are not only more visually convincing but also offer a level of interpretability and user control that pixel-based methods cannot match. Future work will explore more complex physical models of paint mixing to further enhance realism.
Summary updated