Style-Structure Disentangled Features and Normalizing Flows for Diverse Icon Colorization

Yuan-kui Li, Yun-Hsuan Lien, Yu-Shuen WangCVPR 2022 · OpenReview

Abstract

We present a colorization network that generates flat-color icons according to given sketches and semantic colorization styles. Our network contains a style-structure disentangled colorization module and a normalizing flow. The colorization module transforms a paired sketch image and style image into a flat-color icon. To enhance network generalization and the quality of icons, we present a pixel-wise decoder, a global style code, and a contour loss to reduce color gradients at flat regions and increase color discontinuity at boundaries. The normalizing flow maps Gaussian vectors to diverse style codes conditioned on the given semantic colorization label. This conditional sampling enables users to control attributes and obtain diverse colorization results. Compared to previous methods built upon conditional generative adversarial networks, our approach enjoys the advantages of both high image quality and diversity. To evaluate its effectiveness, we compared the flat-color icons generated by our approach and recent colorization and image-to-image translation methods on various conditions. Experiment results verify that our method out- performs state-of-the-arts qualitatively and quantitatively.

Introduction

Colorizing icons is a distinct challenge compared to colorizing photos or comics. In icon design, structure is often defined not by black lines, but by sharp transitions (discontinuities) between different flat color regions. Existing methods, largely based on conditional Generative Adversarial Networks (c-GANs), frequently struggle with two primary issues: a lack of "flatness" (producing unwanted gradients or smearing) and limited diversity due to mode collapse. This paper introduces a framework that abandons the GAN paradigm in favor of a controlled supervised approach using style-structure disentanglement and Normalizing Flows.

Methodology

The proposed architecture is designed to enforce the sharp boundaries and flat regions essential for professional icon aesthetics.

Style-Structure Disentanglement and Pixel-Wise Decoding

The network utilizes two separate encoders: a structure encoder ($E_c$) to capture shape and a style encoder ($E_s$) to capture color palettes. A key innovation is the Pixel-Wise Decoder, which employs $1 \times 1$ convolutions. By processing each pixel embedding independently, the decoder is architecturally prevented from blurring colors across neighbors. This forces the structure encoder to do the heavy lifting of identifying precise region boundaries.

The proposed architecture featuring style-structure disentanglement and a conditional normalizing flow for diverse icon colorization.

Contour Extraction and Normalizing Flows

To ensure sharp boundaries, the system uses a contour extractor with "NormConv" layers. These layers feature kernels that sum to zero, forcing the network to focus on color gradients (edges) rather than absolute color values. To address the diversity problem, the authors integrate Conditional Continuous Normalizing Flows (c-CNF). This component maps simple Gaussian noise to complex style codes based on 85 semantic labels from the Kobayashi Color Image Scale (e.g., "Luxury," "Sweet"), allowing for diverse, high-quality colorization from a single sketch.

Key Results

The model was evaluated on a dataset of 12,575 icons. Quantitatively, the proposed method achieved a Fréchet Inception Distance (FID) of 27.96, significantly outperforming the next best baseline, AdvIcon, which scored 40.86. It also dominated in metrics for flatness and structural accuracy.

Qualitatively, as shown in the comparison below, the method is the only one that consistently produces truly flat colors without artifacts in small or thin areas. In a user study with 126 participants, the proposed method received a score of 905, dwarfing the highest-scoring baseline's score of 192.

Qualitative comparison showing the proposed method's ability to maintain flat colors and sharp boundaries compared to state-of-the-art baselines.

Conclusion and Future Work

By combining architectural constraints like the pixel-wise decoder with the stability of Normalizing Flows, this framework successfully generates high-quality, diverse, flat-color icons. While highly effective, future work will focus on eliminating slight gradients in exceptionally large regions and extending the model to maintain temporal coherence in icon animations.

Summary updated

Abstract

We present a colorization network that generates flat-color icons according to given sketches and semantic colorization styles. Our network contains a style-structure disentangled colorization module and a normalizing flow. The colorization module transforms a paired sketch image and style image into a flat-color icon. To enhance network generalization and the quality of icons, we present a pixel-wise decoder, a global style code, and a contour loss to reduce color gradients at flat regions and increase color discontinuity at boundaries. The normalizing flow maps Gaussian vectors to diverse style codes conditioned on the given semantic colorization label. This conditional sampling enables users to control attributes and obtain diverse colorization results. Compared to previous methods built upon conditional generative adversarial networks, our approach enjoys the advantages of both high image quality and diversity. To evaluate its effectiveness, we compared the flat-color icons generated by our approach and recent colorization and image-to-image translation methods on various conditions. Experiment results verify that our method out- performs state-of-the-arts qualitatively and quantitatively.
BibTeX
@inproceedings{li2022style,
  author = {Yuan-kui Li and Yun-Hsuan Lien and Yu-Shuen Wang},
  title = {{Style-Structure Disentangled Features and Normalizing Flows for Diverse Icon Colorization}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2022},
  url = {https://openreview.net/forum?id=4wvFYijESR},
}
Figure