TL;DR
This paper introduces SAPGAN, the first end-to-end generative adversarial network (GAN) capable of creating original Chinese landscape paintings without human input. By decoupling sketch generation from painting, SAPGAN produces high-resolution, aesthetically pleasing artworks that are often mistaken for human creations, paving the way for truly machine-original art.
What Problem Does This Paper Address?
Current generative art models often rely heavily on conditional inputs, such as existing sketches or photographs, limiting their originality and creativity. While GANs have shown promise in style transfer and image generation, research in East Asian art, particularly traditional Chinese landscape painting, has been relatively overlooked. Furthermore, existing methods often produce derivative works that are mere stylistic copies rather than truly novel creations. This paper aims to bridge this gap by developing a GAN that can generate original Chinese landscape paintings from scratch, mimicking the human creative process of sketching first and then painting.
Methods
The proposed framework, Sketch-And-Paint GAN (SAPGAN), is a two-stage process:
- Stage I: SketchGAN: This stage generates high-resolution edge maps that serve as the structural "sketches" for the paintings. The authors experiment with two GAN architectures for this stage:
- RaLSGAN: A relativistic GAN known for stable training and high-quality image synthesis.
-
StyleGAN2: A state-of-the-art unconditional GAN for generating high-resolution images, chosen for its improved quality and artifact reduction. These GANs are trained on HED (Holistically-Nested Edge Detection) edge maps derived from the dataset.
-
Stage II: PaintGAN: This stage takes the generated sketches from SketchGAN as input and performs an image-to-image translation to produce the final painting. The authors evaluate several conditional GAN architectures for this stage:
- Pix2Pix: A standard image-to-image translation model.
- Pix2PixHD: An advanced version for high-resolution, photorealistic synthesis.
- SPADE: A state-of-the-art model that uses spatially-adaptive normalization to improve semantic image synthesis.
The overall SAPGAN framework combines these stages, allowing for end-to-end generation from a latent vector.
Datasets
The authors introduce a new dataset of 2,192 high-quality traditional Chinese landscape paintings. These paintings were collected from open-access museum collections (Smithsonian Freer Gallery, Metropolitan Museum of Art, Princeton University Art Museum, and Harvard University Art Museum). The dataset was curated by manually filtering out non-landscape artworks and cleaning up calligraphy or silk borders. Paintings were then cropped and resized to 512x512 pixels. HED edge maps were generated from these paintings for training the SketchGAN stage.
Key Insights and Ideas
- Decoupling Sketch and Paint: The core innovation is separating the generation process into two distinct stages: structural sketch generation and detail-infused painting. This mimics human artistic creation and allows for more control and originality.
- Unsupervised Art Generation: By generating from a latent space rather than relying on direct human input for each painting, SAPGAN moves towards truly machine-original art.
- HED Edge Maps as Sufficient Input: The research demonstrates that HED edge maps, which capture high-level shapes and some low-level details, are sufficient for generating realistic landscape paintings, making the approach more generalizable than methods requiring complex input.
- Cultural Specificity: Focusing on traditional Chinese landscape paintings addresses a gap in GAN research, which has historically favored Western art.
- Robustness to Overfitting: The SAPGAN models show robustness to overfitting, meaning they can creatively deviate from their training data, a crucial aspect for genuine originality.
Key Results
- Visual Quality: SAPGAN paintings, particularly those using RaLSGAN+Pix2Pix and StyleGAN2+Pix2Pix configurations, are described as more edge-defined and realistic than baseline GANs. They exhibit structural composition and well-defined landscapes, sometimes even capturing atmospheric effects like fading perspective.
- Human Study (Visual Turing Test):
- Out of 242 participants, SAPGAN paintings were mistaken for human-created art 55% of the time, significantly outperforming baseline RaLSGAN (11% mistaken for human).
- SAPGAN paintings were rated consistently higher than baseline models across artistic categories: "aesthetically pleasing," "artfully-composed," "clear," and "creativity" (all with p < 0.0001).
- Nearest Neighbor Test: SAPGAN models demonstrated a better ability to deviate from their training data compared to baseline models, indicating less memorization and more creative generation.
Limitations and Future Work
- Limitations:
- While SAPGAN excels at landscapes, it struggles with generating Chinese calligraphy, which was not the primary focus.
- SPADE, a promising PaintGAN candidate, showed signs of over-fitting with oversaturated colors, leading the authors to exclude it from further testing in that specific configuration.
- The qualitative ratings for SAPGAN were still lower than human artwork, indicating room for improvement in realism.
- Future Work:
- Exploring different GAN architectures for both SketchGAN and PaintGAN to enhance functionality.
- Developing multimodal generation capabilities for different painting styles.
- Incorporating brushstrokes or calligraphy into generated paintings to increase authenticity.
- Generalizing the SAPGAN framework to other artistic styles that emphasize edge definition, beyond Chinese paintings.
Summary updated