Learnable Fractal Flames

Jordan J. Bannister, Derek NowrouzezahraiarXiv 2024 · OpenReview

TL;DR

This paper introduces a novel differentiable rendering pipeline for fractal flames, enabling artists to guide fractal generation using reference images through gradient descent optimization. This approach extends previous work by supporting color images, non-linear generator functions, and the composition of multiple fractals, opening new avenues for intuitive and expressive fractal art creation.

What Problem Does This Paper Address?

Creating visually appealing fractal art traditionally involves a laborious trial-and-error process of tweaking mathematical parameters. Artists lack an intuitive way to guide the generation of fractals to match a desired aesthetic or a specific reference image. While prior work has explored inverse fractal rendering, these methods often focused on image compression or were limited to simpler, linear fractal generation and grayscale output. This paper tackles the challenge of enabling artists to "learn" fractal parameters from a target image, making the creative process more direct and accessible, particularly for complex, colorful fractal flames.

Methods

The core contribution is a differentiable rendering pipeline for fractal flames. This pipeline allows gradients to flow from a target image back to the fractal's latent parameters, enabling optimization via gradient descent. The pipeline consists of four key differentiable components:

  • Sampler: Generates sample positions and "quality vectors" based on a set of differentiable generator functions. Crucially, it separates the random sampling of generator order from the sample generation itself to allow for gradient flow.
  • Splatter: "Splatts" the quality vectors onto an image buffer at the sample positions. This process uses a Gaussian kernel and is designed to be differentiable, relating pixel values continuously to sample positions.
  • Painter: Transforms the splatted quality vectors into RGBA color values for each pixel. This module incorporates learnable parameters and uses a log-density tone mapping approach, similar to prior fractal rendering techniques, but made differentiable.
  • Compositor: Allows for the combination of multiple fractal flames, each with its own parameters, over a learnable background color. This enables the creation of more complex and layered artwork.

The entire pipeline is implemented using the Taichi language, which supports automatic differentiation and GPU acceleration.

Datasets

The paper uses existing artworks as reference images for training and evaluation. Specifically, they mention:

  • Simple reference images: Used for demonstrating the learning of non-linear fractal flames (Figure 4).
  • Famous paintings:
    • "Fighting Forms" by Franz Marc (1914)
    • "Wisteria" by Claude Monet (1925) These paintings were used for training multi-fractal compositions.

No new datasets were introduced; instead, the approach leverages existing visual art as targets for the learning process.

Key Insights and Ideas

  • Differentiable Fractal Flames: The ability to backpropagate gradients through the fractal rendering process is the central innovation, transforming fractal art creation from a manual tuning process to an optimization problem.
  • Non-linear Generator Functions: Supporting non-linear functions significantly expands the aesthetic possibilities of fractal flames, allowing for much richer and more varied visual outputs compared to linear IFS fractals.
  • Multi-Fractal Compositions: The differentiable compositor allows for the combination of multiple fractal flames, enabling artists to layer and blend different fractal styles to achieve greater complexity and fidelity to reference images.
  • Sensitivity to Initialization: The loss landscape for fractal optimization is non-convex, meaning that different initial parameter values can lead to distinct local minima and visually different fractal outputs. This offers artists a way to explore a range of aesthetic outcomes by manipulating initial conditions.
  • Resolution Matters: The resolution of the training image impacts the splatting kernel and, consequently, the gradients used during optimization. Lower resolutions tend to produce fractals with sharper, thinner structures.

Key Results

  • The proposed approach successfully learned fractal flame parameters from reference images, producing visually similar results.
  • The system can generate complex and colorful fractal artwork, including compositions based on famous paintings like those by Franz Marc and Claude Monet.
  • Experiments demonstrated that increasing the number of generator functions allows for the representation of more complex patterns.
  • The system can achieve good results in approximately 20 minutes for computationally expensive multi-flame training on an Nvidia 2070 Super GPU.

Limitations and Future Work

  • Training Instability: While not a frequent issue in their experiments, the authors acknowledge potential training instability when using non-linear generator functions.
  • Stochastic Generator Functions: The current implementation does not support stochastic generator functions, which are present in some fractal flame variations. Supporting these would require further application of the re-parameterization trick.
  • Interactive Applications: The authors suggest integrating this differentiable rendering approach into interactive graphical applications. This would allow artists without programming backgrounds to easily access and explore the technology, similar to existing fractal flame creation software.

Summary updated

Abstract

This work presents a differentiable rendering approach that allows latent fractal flame parameters to be learned from image supervision using gradient descent optimization. The approach extends the state-of-the-art in differentiable iterated function system fractal rendering through support for color images, non-linear generator functions, and multi-fractal compositions. With this approach, artists can use reference images to quickly and intuitively control the creation of fractals. We describe the approach and conduct a series of experiments exploring its use, culminating in the creation of complex and colorful fractal artwork based on famous paintings.
BibTeX
@misc{bannister2024learnable,
  author = {Jordan J. Bannister and Derek Nowrouzezahrai},
  title = {{Learnable Fractal Flames}},
  year = {2024},
  eprint = {2406.09328},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2406.09328},
}

Loading related content...

Figure