ZM-Net: Real-time Zero-shot Image Manipulation Network

Hao Wang, Xiaodan Liang, Hao Zhang, Dit-Yan Yeung, Eric P. XingarXiv 2017 · OpenReview

Abstract

ZM-Net is a unified deep learning framework designed for real-time image manipulation, such as style transfer and attribute editing. It uniquely generalizes to unseen guiding signals without the need for retraining. By employing a Parameter Network to dynamically generate transformation parameters for a Transformation Network, ZM-Net can handle over 20,000 styles within a single model and perform zero-shot manipulations in milliseconds.

Introduction

Image manipulation—transforming a content image based on a guiding signal like an art painting or a text description—has traditionally faced a significant "speed vs. flexibility" trade-off. Optimization-based methods offer high flexibility but are computationally expensive, often taking minutes per image. Conversely, feed-forward CNNs provide real-time performance but are typically limited to a "one-network-per-style" paradigm, requiring hours of retraining for every new style.

ZM-Net addresses this gap by providing a single, real-time network capable of zero-shot manipulation. This means the model can apply a style or attribute it has never encountered during training, instantly.

Methods

The core innovation of ZM-Net lies in its decoupled architecture, consisting of two primary sub-networks:

  • PNet (Parameter Network): This serves as the encoder for the guiding signal. It processes a style image or a word embedding (e.g., the vector for "night") to generate key transformation parameters.
  • TNet (Transformation Network): This network processes the content image. It utilizes Dynamic Instance Normalization (DIN), where the scaling ($\gamma$) and shifting ($\beta$) factors are generated on-the-fly by the PNet rather than being learned constants. This allows the TNet to fundamentally adapt its behavior for every unique input style.
The ZM-Net architecture featuring the Transformation Network (TNet) and the Parameter Network (PNet) guided by a signal.

The authors found that a Serial PNet—a deep CNN mirroring the TNet's structure—produced higher-quality results with fewer artifacts compared to parallel alternatives.

Datasets and Training

To achieve robust generalization, the model was trained on a diverse set of data:

  • Content: MS-COCO dataset.
  • Styles: 984 impressionist paintings from the Pandora dataset were expanded into a massive dataset of 23,307 style images. This was achieved by using slow optimization-based methods to transfer seed styles onto diverse content, preventing the model from overfitting to specific textures.
  • Attributes: For semantic manipulation, the authors utilized images tagged "noon" and "night" to learn transitions between different times of day.

Key Results and Insights

ZM-Net demonstrates remarkable efficiency and scalability, processing images in 15ms to 38ms, making it suitable for real-time video applications. While previous state-of-the-art models were limited to dozens of styles, ZM-Net successfully incorporates over 20,000 styles into a single model.

Zero-shot style transfer and semantic attribute editing (day-to-night) results.

Key Insights:

  • Decoupling Style from Content: By isolating style information within normalization parameters, the network effectively separates structural content from stylistic appearance.
  • Cross-Modal Flexibility: The PNet's ability to process vectorized inputs allows ZM-Net to perform semantic edits using word embeddings. For instance, when guided by the word "night," the model learns to "turn on" lights in building windows rather than just darkening the image.
  • Zero-Shot Generalization: The PNet learns a mapping from style space to parameter space, allowing it to interpolate and generate appropriate parameters for paintings it has never seen before.

Conclusion

ZM-Net represents a significant step forward in real-time image manipulation. While the Serial PNet introduces some computational overhead, it remains well within real-time limits. Future work may explore extending this framework to other modalities, such as audio-guided manipulation, or refining the PNet to handle increasingly abstract semantic attributes.

Summary updated

Abstract

Many problems in image processing and computer vision (e.g. colorization, style transfer) can be posed as 'manipulating' an input image into a corresponding output image given a user-specified guiding signal. A holy-grail solution towards generic image manipulation should be able to efficiently alter an input image with any personalized signals (even signals unseen during training), such as diverse paintings and arbitrary descriptive attributes. However, existing methods are either inefficient to simultaneously process multiple signals (let alone generalize to unseen signals), or unable to handle signals from other modalities. In this paper, we make the first attempt to address the zero-shot image manipulation task. We cast this problem as manipulating an input image according to a parametric model whose key parameters can be conditionally generated from any guiding signal (even unseen ones). To this end, we propose the Zero-shot Manipulation Net (ZM-Net), a fully-differentiable architecture that jointly optimizes an image-transformation network (TNet) and a parameter network (PNet). The PNet learns to generate key transformation parameters for the TNet given any guiding signal while the TNet performs fast zero-shot image manipulation according to both signal-dependent parameters from the PNet and signal-invariant parameters from the TNet itself. Extensive experiments show that our ZM-Net can perform high-quality image manipulation conditioned on different forms of guiding signals (e.g. style images and attributes) in real-time (tens of milliseconds per image) even for unseen signals. Moreover, a large-scale style dataset with over 20,000 style images is also constructed to promote further research.
BibTeX
@misc{wang2017zm,
  author = {Hao Wang and Xiaodan Liang and Hao Zhang and Dit-Yan Yeung and Eric P. Xing},
  title = {{ZM-Net: Real-time Zero-shot Image Manipulation Network}},
  year = {2017},
  eprint = {1703.07255},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1703.07255},
}
Figure