Cloud2Sketch: Augmenting Clouds with Imaginary Sketches

Zhaoyi Wan, Dejia Xu, Zhangyang Wang, Jian Wang, Jiebo Luo

Abstract

Have you ever looked up at the sky and imagined what the clouds look like? In this work, we present an interesting task that augments clouds in the sky with imagined sketches. Different from generic image-to-sketch translation tasks, unique challenges are introduced: real-world clouds have different levels of similarity to something; sketch generation without sketch retrieval could lead to something unrecognizable; a retrieved sketch from some dataset cannot be directly used because of the mismatch of the shape; an optimal sketch imagination is subjective. We propose Cloud2Sketch, a novel self-supervised pipeline to tackle the aforementioned challenges. First, we pre-process cloud images with a cloud detector and a thresholding algorithm to obtain cloud contours. Then, cloud contours are passed through a retrieval module to retrieve sketches with similar geometrical shapes. Finally, we adopt a novel sketch translation model with built-in free-form deformation for aligning the sketches to cloud contours. To facilitate training, an icon-based sketch collection named Sketchy Zoo is proposed. Extensive experiments validate the effectiveness of our method both qualitatively and quantitatively.
BibTeX
@inproceedings{wan2022cloud,
  author = {Zhaoyi Wan and Dejia Xu and Zhangyang Wang and Jian Wang and Jiebo Luo},
  title = {{Cloud2Sketch: Augmenting Clouds with Imaginary Sketches}},
  booktitle = {ACM Multimedia},
  year = {2022},
  doi = {10.1145/3503161.3547810},
}

Loading related content...

Figure