One-Shot Reference-based Structure-Aware Image to Sketch Synthesis

Rui Yang, Honghong Yang, Li Zhao, Qin Lei, Mianxiong Dong, Kaoru Ota, Xiaojun WuAAAI 2025 · DOI

TL;DR

This paper introduces Ref2Sketch-SA, a novel training-free method for generating sketches from a content image guided by a reference sketch. It effectively balances preserving the content's structure with adopting the reference sketch's artistic style, overcoming limitations of existing methods that either require extensive training data or fail to capture diverse sketch styles.

What Problem Does This Paper Address?

Generating sketches that accurately capture the essence of a reference sketch while retaining the structure of a content image is a significant challenge. Existing methods often fall into two camps: those requiring large, paired datasets for training, limiting their adaptability to new styles, and those that are "zero-shot" but struggle to disentangle content from style or fail to capture the nuanced artistic variations present in hand-drawn sketches. Traditional edge detection methods are too simplistic, and even recent diffusion model-based approaches like InstantStyle can lose stylistic elements or require additional constraints like ControlNet, which can dilute the style intensity. The core problem is achieving a faithful, one-shot transfer of artistic style from a reference sketch to a content image without extensive training or compromising structural integrity.

Methods

Ref2Sketch-SA leverages a pre-trained diffusion model (SDXL) and introduces several key innovations:

  • DDIM Inversion: This technique is used to enhance structural consistency by processing the content image in the latent space, ensuring that the generated sketch's layout closely matches the original.
  • Texture Enhanced Module: This module injects adaptive noise into the image during the denoising process. It uses high-frequency and low-frequency maps derived from the content image to selectively blend noise, preserving fine details (edges) while allowing for more noise in broader regions to facilitate detail generation and refinement. This helps in capturing the texture and line styles of the reference sketch.
  • Style Adapter (IP-Adapter based): This component is crucial for style transfer. It injects features from the reference sketch into specific attention layers of the diffusion model. By targeting layers that handle style and spatial layout, it effectively transfers the reference sketch's style (brushstrokes, line thickness, texture) without causing "content leakage" from the reference image itself.
  • Acutance Regularization: To ensure sharpness and detail in the generated sketches, a regularization loss based on gradient magnitudes (using a Sobel operator) is applied. This encourages the model to produce clear edges and structural details, enhancing the sketch's fidelity.
  • Denoising Correction: A final revision step is incorporated into the denoising process to further refine the output based on the regularization.

Datasets

The paper evaluates its method on several datasets:

  • 4SKST: Used for comparing sketch extraction methods.
  • Anime: Used for evaluating style transfer on anime sketches.
  • FS2K: Used for ablation studies.
  • APDrawings: Used for visual comparison with other sketch generation models.

The authors note that their method is training-free and does not require new benchmarks.

Key Insights and Ideas

  • Sketching as a Specialized Style Transfer: The paper frames sketch generation as a unique form of style transfer, emphasizing the importance of both content preservation and style adoption.
  • Adaptive Noise Blending for Texture: The idea of using frequency maps to guide noise injection is a clever way to balance detail preservation with style transfer, allowing different parts of the image to be treated differently based on their structural importance.
  • Targeted Style Injection: Injecting style features into specific attention layers (rather than globally) is key to achieving style transfer without content leakage and maintaining structural integrity. This highlights the importance of understanding the internal workings of diffusion models for fine-grained control.
  • Training-Free Advantage: The ability to achieve state-of-the-art results without any training significantly lowers the barrier to entry for users wanting to generate sketches in custom styles.

Key Results

  • Superior Qualitative Results: Visual comparisons (Figure 4 and 5) show that Ref2Sketch-SA produces sketches that are more faithful to the reference style and content structure compared to existing training-based and training-free methods. It successfully captures nuanced styles that other methods miss.
  • Quantitative Performance: Tables 2 and 3 show that Ref2Sketch-SA achieves the best scores across LPIPS, PSNR, and FID metrics on the 4SKST and Anime datasets, outperforming specialized sketch extraction methods and general image-to-image translation models.
  • User Study Dominance: A user study with 270 participants overwhelmingly preferred sketches generated by Ref2Sketch-SA, with the method receiving 59.01% of the votes, significantly outperforming all other tested methods.
  • Ablation Study Validation: The ablation study (Table 4) confirms that each component of Ref2Sketch-SA (Inversion, Texture Enhanced Module, Acutance Regularization) contributes to the overall performance, with the full model yielding the best results.

Limitations and Future Work

  • Manual Parameter Adjustment: The paper acknowledges that the parameter $\beta$, which controls the sparsity of the binary image, requires manual adjustment.
  • Future Work Suggestion: The authors suggest that future work could explore automating the adjustment of $\beta$ based on the style of the reference image, which would further enhance the model's adaptability and ease of use.

Summary updated

BibTeX
@inproceedings{yang2025one,
  author = {Rui Yang and Honghong Yang and Li Zhao and Qin Lei and Mianxiong Dong and Kaoru Ota and Xiaojun Wu},
  title = {{One-Shot Reference-based Structure-Aware Image to Sketch Synthesis}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.1609/aaai.v39i9.33000},
}
Figure