Vector Image Generation by Learning Parametric Layer Decomposition

Othman Sbai, Camille Couprie, Mathieu Aubry

Introduction

Modern deep generative models, such as GANs and VAEs, typically represent images as discrete grids of pixels. While effective, this approach suffers from a lack of interpretability—making it difficult to isolate and edit specific image components—and resolution constraints, where scaling leads to pixelation. Researchers from Facebook AI Research and École des Ponts address these limitations by proposing an unsupervised method to decompose images into a stack of colored "vector" layers. By representing image masks as continuous parametric functions, the system enables infinite-resolution rendering and intuitive, layer-based editing without requiring manual labels.

Methodology

The core of the proposed approach is an iterative "painting" process. Rather than generating an entire image in a single pass, the model constructs it layer by layer, typically over 10–20 steps.

The iterative layer-by-layer painting process used to reconstruct images.

At each step $t$, a ResNet encoder processes the target image and the current state of the canvas to predict two primary outputs: a single RGB color ($c_t$) and a set of parameters ($p_t$) for a transparency mask. The "vector" engine is a small Multi-Layer Perceptron (MLP) that acts as a continuous mask generator. Given a coordinate $(x, y)$ and the predicted parameters, the MLP computes the transparency $\alpha$ at that point. This mathematical representation allows the mask to be queried at any resolution. Finally, the new colored mask is blended onto the canvas using standard alpha composition. The system is trained end-to-end using a combination of $L_1$ loss, perceptual loss (VGG), and adversarial loss to ensure high-quality reconstructions.

Experiments and Results

The model was evaluated on three diverse benchmarks: CelebA (faces), ImageNet (natural objects), and MNIST (digits).

  • Reconstruction Quality: On the CelebA dataset, the model achieved a PSNR of 24.02, outperforming existing vector-based baselines.
  • Resolution Independence: MNIST digits trained at 28x28 resolution remained perfectly sharp when rendered at 1024x1024, demonstrating the power of the coordinate-based representation compared to standard bilinear upsampling.
  • Attribute Retrieval: The semantic nature of the learned mask parameters ($p_t$) allows for advanced image searches. For instance, searching for "people wearing hats" based on mask similarity yielded a 34% average precision in a one-shot setting, significantly outperforming standard feature-based methods.
  • Interactive Editing: The authors demonstrated a UI where users can select image regions (like hair) to instantly modify color or shape, facilitated by the layer-based decomposition.

Key Insights and Discussion

A significant finding of this work is the emergent semantics within the unsupervised framework. Despite receiving no explicit labels for object parts, the model naturally learns to assign different layers to distinct semantic regions, such as hair, skin, or background.

An emergent semantic mask capturing the hair region of a face.

The authors also noted that a recursive prediction strategy is superior to one-shot generation. By observing the current canvas before predicting the next layer, the model makes more informed decisions about stroke placement. However, challenges remain regarding computational speed and the representation of complex textures with solid-color layers.

Conclusion

By bridging the gap between deep generative modeling and vector graphics, this work provides a path toward more interpretable and scalable image synthesis. The ability to decompose real-world images into editable, resolution-independent layers opens new possibilities for interactive design and semantic image understanding.

Summary updated

BibTeX
@inproceedings{sbai2020vector,
  author = {Othman Sbai and Camille Couprie and Mathieu Aubry},
  title = {{Vector Image Generation by Learning Parametric Layer Decomposition}},
  booktitle = {ICIP},
  year = {2020},
  doi = {10.1109/icip40778.2020.9190638},
}
Figure