Making Robots Draw A Vivid Portrait In Two Minutes

Fei Gao, Jingjie Zhu, Zeyuan Yu, Peng Li, Tao Wang

TL;DR

This paper introduces AiSketcher, a robotic system that can generate high-quality facial portraits in under two minutes using a novel deep learning-based portrait synthesis algorithm. By combining self-consistency and compositional sparsity losses with a robotic arm, AiSketcher achieves a balance between artistic detail and rapid execution, outperforming existing methods.

What Problem Does This Paper Address?

While artistic robots have made strides, a significant challenge remains in producing high-quality portraits quickly. Existing robots often struggle with faces in challenging poses or with occlusions, and their drawing times can be prohibitively long (hours for colorful images). Furthermore, achieving a balance between the vividness of a sketch (requiring detail) and the time budget (favoring simplicity) is a core difficulty. This paper aims to bridge this gap by developing a system that can automatically generate and draw vivid portraits in a matter of minutes.

Methods

The core of AiSketcher is a novel portrait synthesis algorithm built upon deep neural networks, inspired by Neural Style Transfer (NST). The system incorporates several key innovations:

  • Global Sketch Synthesis: This module uses a base NST algorithm but enhances it with two novel loss functions:
  • Self-consistency Loss: This encourages the algorithm to reconstruct real sketches when given a sketch as input, leading to more continuous and realistic brush strokes.
  • Componential Sparsity Loss: This constraint reduces the number of brush strokes in less significant areas (like backgrounds or skin) by leveraging face parsing masks, thus speeding up the drawing process without sacrificing portrait quality.
  • Local Sketch Synthesis: To address occasional poor generation of eyebrows, a separate GAN-based module is used specifically for synthesizing eyebrows.
  • Pre- and Post-processing: The system includes face detection, alignment, and parsing for initial image preparation. Post-processing steps like image binarization and fusing global and local sketches are used to refine the final output.
  • Robotic Arm Integration: A 3-DOF robotic arm (uArm) is used to reproduce the synthesized sketch on paper. The synthesized sketch is converted into a sequence of trajectories for the robot to follow.

Datasets

  • Content Images: 1,000 low-resolution facial pictures from the CelebA dataset and 1,000 high-resolution facial pictures from the CelebA-HQ dataset were used for training.
  • Style Images: 20 real sketches of arbitrary content were used as style images.
  • Eyebrow Pairs: 280 eyebrow photo-sketch pairs were used to train the local sketch synthesis algorithm.
  • Testing: Approximately 200 images, including celebrity faces and universal images, were used for testing.

Key Insights and Ideas

  • NST for Robotics is Novel: The paper highlights that AiSketcher is the first portrait drawing robot to leverage Neural Style Transfer techniques, opening up new avenues for robotic art generation.
  • Balancing Vividness and Speed: The core challenge of creating detailed yet quickly drawn portraits is addressed by the dual approach of generating rich details where needed (e.g., facial features) and simplifying where possible (e.g., background).
  • Self-consistency for Smoothness: The self-consistency loss is crucial for achieving smooth, continuous brush strokes, which are essential for a natural-looking sketch.
  • Componential Sparsity for Efficiency: By intelligently sparsifying less critical regions, the system significantly reduces drawing time without degrading the overall quality, a key differentiator from methods that might over-simplify.
  • Modular Approach: The combination of global and local synthesis, along with pre- and post-processing, allows for targeted improvements and robustness across different facial features and image types.

Key Results

  • Speed: AiSketcher can synthesize a sketch in approximately 1 second (on a server with a GPU) and draw it in 1-3 minutes, achieving an average total drawing time of under two minutes.
  • Quality: The system produces considerably high-quality sketches that capture individual characteristics. It handles faces in-the-wild, including diverse poses, expressions, and occlusions, outperforming existing methods like APDrawingGAN in terms of detail and stroke quality.
  • Generalization: AiSketcher also demonstrates good performance on universal images (non-facial), producing high-quality sketches unless the image has very delicate textures or is low-resolution.
  • Ablation Studies: Experiments confirm the effectiveness of the proposed self-consistency loss (improving smoothness) and compositional sparsity loss (reducing brush strokes without sacrificing quality).

Limitations and Future Work

  • Delicate Elements: The authors acknowledge that the 3-DOF robotic arm has limitations in precisely reproducing very fine details present in the synthesized sketches.
  • Undesired Brush Strokes: The sketch synthesis algorithm can sometimes produce undesired brush strokes along shadow boundaries. Illumination normalization is suggested as a potential solution.
  • Future Directions:
    • Optimizing path planning and portrait synthesis algorithms to better account for robotic arm limitations.
  • Exploring visual feedback for more accurate reproduction.
  • Improving drawing speed and energy efficiency by developing robot-specific processors.
  • Investigating illumination normalization to address artifacts in shadow regions.

Summary updated

Abstract

Significant progress has been made with artistic robots. However, existing robots fail to produce high-quality portraits in a short time. In this work, we present a drawing robot, which can automatically transfer a facial picture to a vivid portrait, and then draw it on paper within two minutes averagely. At the heart of our system is a novel portrait synthesis algorithm based on deep learning. Innovatively, we employ a self-consistency loss, which makes the algorithm capable of generating continuous and smooth brush-strokes. Besides, we propose a componential sparsity constraint to reduce the number of brush-strokes over insignificant areas. We also implement a local sketch synthesis algorithm, and several pre- and post-processing techniques to deal with the background and details. The portrait produced by our algorithm successfully captures individual characteristics by using a sparse set of continuous brush-strokes. Finally, the portrait is converted to a sequence of trajectories and reproduced by a 3-degree-of-freedom robotic arm. The whole portrait drawing robotic system is named AiSketcher. Extensive experiments show that AiSketcher can produce considerably high-quality sketches for a wide range of pictures, including faces in-the-wild and universal images of arbitrary content. To our best knowledge, AiSketcher is the first portrait drawing robot that uses neural style transfer techniques. AiSketcher has attended a quite number of exhibitions and shown remarkable performance under diverse circumstances.
BibTeX
@misc{gao2020making,
  author = {Fei Gao and Jingjie Zhu and Zeyuan Yu and Peng Li and Tao Wang},
  title = {{Making Robots Draw A Vivid Portrait In Two Minutes}},
  year = {2020},
  eprint = {2005.05526},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2005.05526},
}

Loading related content...

Figure