InstaStyle: Inversion Noise of a Stylized Image is Secretly a Style Adviser

Xing Cui, Zekun Li, Pei-Pei Li, Huaibo Huang, Zhaofeng He

Introduction

Text-to-image diffusion models have transformed digital art, yet precise "stylization"—the ability to generate new content that mimics the specific brushstrokes, textures, and color palettes of a single reference image—remains difficult. Existing methods often suffer from prompt ambiguity, where language fails to capture artistic nuances, and content leakage, where the model accidentally copies objects from the reference image instead of just its style.

InstaStyle addresses these hurdles by revealing that the noise extracted from a stylized image via DDIM inversion is not merely random. Instead, it acts as a "style adviser" that carries fine-grained artistic signals, enabling high-fidelity stylization from a single reference.

The "Secret" Style Adviser

The core innovation of InstaStyle lies in the observation that DDIM inversion (reversing an image back into noise) results in a latent representation with a non-zero Signal-to-Noise Ratio (SNR). At the terminal timestep ($T=1000$), this "style noise" remains biased toward the reference's structural and textural properties. By using this biased noise as a starting point for new generations rather than standard Gaussian noise, the model receives a structural guide that preserves the reference's artistic essence.

Methodology: A Two-Stage Framework

InstaStyle operates in two distinct stages to bridge the gap between coarse text prompts and fine-grained styles:

  1. Noise Extraction & Initial Generation: The system performs DDIM inversion on the reference image to extract the "style adviser" noise. This noise is then used to initialize the diffusion process for new target prompts.
  2. Prompt Refinement (Human-in-the-Loop): To capture nuances language cannot describe, the authors introduce a learnable style token (e.g., <style1>). The model generates a set of initial images, and a human selector identifies the top $N$ examples that best represent the desired style. The model then uses Low-Rank Adaptation (LoRA) to fine-tune the style token and cross-attention layers based on these high-quality selections.

Additionally, InstaStyle supports Style Combination, allowing users to blend two distinct artistic styles by masking and mixing their respective inversion noises and interpolating between learned style tokens.

Key Results and Evaluation

InstaStyle demonstrates superior performance across both quantitative and qualitative metrics. Using CLIP-based evaluations on 100 object classes from CIFAR-100, the framework achieved a Style Consistency score of 0.655 and a Content Fidelity score of 0.294, outperforming established baselines like StyleDrop and DreamBooth.

InstaStyle results showing stylized image generation, style combination, and interpolation.

Qualitative results highlight the model's ability to preserve specific artistic details—such as the grain of watercolor paper or the thickness of oil paint—without "copy-pasting" content from the source. In user studies, InstaStyle was preferred over Textual Inversion and Custom Diffusion more than 75% of the time.

Limitations and Future Work

While highly effective, the model occasionally struggles with intricate mechanical or biological details, such as complex gears or delicate insect wings. Furthermore, the current reliance on human selection for prompt refinement presents an opportunity for future automation via aesthetic or style-scoring models. As diffusion backbones continue to evolve, the InstaStyle framework is poised to provide even higher levels of detail and artistic control.

Summary updated

BibTeX
@inproceedings{cui2024instastyle,
  author = {Xing Cui and Zekun Li and Pei-Pei Li and Huaibo Huang and Zhaofeng He},
  title = {{InstaStyle: Inversion Noise of a Stylized Image is Secretly a Style Adviser}},
  booktitle = {European Conference on Computer Vision},
  year = {2024},
  doi = {10.1007/978-3-031-72983-6\_26},
}
Figure