SketchTriplet: Self-Supervised Scenarized Sketch–Text–Image Triplet Generation

Zhenbei Wu, Qiang Wang, Jie Yang

TL;DR

This paper introduces SketchTriplet, a self-supervised method to generate complex scene sketches from simple single-object sketches. It tackles the scarcity of large-scale scene sketch datasets by proposing a novel approach to expand semantic information and generate "sketch-text-image" triplets, achieving state-of-the-art results in zero-shot image-to-sketch tasks and providing a valuable new dataset for the community.

What Problem Does This Paper Address?

Creating detailed scene sketches is challenging, and existing datasets primarily focus on single objects, lacking the complexity and richness of real-world scenes. This scarcity hinders the development of advanced AI models for tasks like sketch-based image retrieval and controlled image synthesis. The paper aims to bridge this gap by enabling the generation of scene sketches from simpler inputs, without requiring pre-existing scene sketch data.

Methods

The core of the proposed method, SketchTriplet, involves a two-stage process:

  1. Sketch-Based Semantic Information Expansion:
    • Vector Sketch Captioning: A Graph Convolutional Network (GCN) is used to extract foundational semantic elements from vector sketches, focusing on object shapes rather than color or texture.
  2. Semantic Expansion: This extracted sketch information is then expanded using a large-scale image captioning dataset and a fusion model (leveraging ChatGPT) to create rich, semantically consistent textual descriptions for scene sketches.
  3. Scene Sketch Generation:
    • Text-Driven Canvas Layout Adaptation: The expanded textual descriptions guide the adaptation of the single-object sketch's layout, adjusting the size and position of the main object to fit a scene context.
  4. Visualization: A latent diffusion model visualizes the expanded semantic information to generate a corresponding image.
  5. Multi-condition Constrained Sketch Generation: A GAN-based model then transforms this generated image into a scene sketch, guided by multiple constraints: semantic fusion perception (CLIP-based alignment with text and image), sketch object content perception (ensuring the original object is preserved), and multi-class object perception (maintaining object layout consistency). This generation process is self-supervised, avoiding the need for paired sketch-image data.

Datasets

  • Training Data:
    • Quickdraw dataset (200 classes, 1,000 sketches each, totaling 20,000 annotated sketch-text pairs) for the sketch captioning module.
  • ADE20K dataset for image captioning to build the semantic expansion repository.
  • A newly constructed dataset, SketchTriplet, comprising 1,000,000 unique scene sketches, each with a corresponding image and semantic label. This dataset is derived from Quickdraw categories and reclassified based on ImageNet-1K categories.

Key Insights and Ideas

  • Leveraging Single-Object Sketches for Scene Generation: The paper demonstrates that rich scene semantics can be effectively "expanded" from simple single-object sketches, overcoming the limitations of existing datasets.
  • Self-Supervised Scene Sketch Generation: By designing a multi-modal fusion and constraint-based generation process, the method achieves scene sketch generation without relying on paired scene sketch-image data, a significant hurdle in previous work.
  • "Sketch-Text-Image" Triplet Generation: The method naturally produces semantically consistent triplets, which are valuable for downstream tasks and contribute to a richer understanding of visual concepts.
  • Importance of Multi-modal Constraints: The ablation studies highlight that combining semantic, object content, and multi-class object perception is crucial for generating high-quality, coherent scene sketches.

Key Results

  • State-of-the-Art Performance: The proposed method achieves state-of-the-art results in zero-shot image-to-sketch generation tasks, outperforming existing methods across various metrics (e.g., FID, IS, LPIPS, CLIPiga) as shown in Table 1 and Figure 3.
  • Improved Downstream Tasks: Retraining existing models on the newly introduced SketchTriplet dataset significantly enhances their performance in sketch-based image retrieval and sketch-controlled image synthesis (demonstrated in Table 2 and Figure 4).
  • Qualitative Improvements: The generated scene sketches are described as complete, clean, and semantically aligned with the input text and original object sketch, as evidenced by qualitative comparisons.

Limitations and Future Work

  • Lack of Transparency Control: The current sketch generation method does not offer control over transparency. The authors suggest that making brush transparency a learnable parameter could address this.
  • Fixed Sketch Style: The method produces sketches in only one fixed style. Future work could focus on enhancing the model's ability to learn and generate sketches in diverse styles.

Summary updated

Abstract

Touchscreen IoT devices, such as smartphones and tablets, have been seamlessly integrated into our daily lives. Drawing sketches on the touch screen is an extremely convenient mode of interaction, and when combined with generative AI, it makes the customized data generation and digital twin based on IoT devices more straightforward. However, the scarcity of free-hand sketch data makes the construction of data-driven generative AI models a thorny issue. Despite the emergence of some large-scale sketch datasets, these datasets primarily consist of sketches at the single-object level. There continues to be a lack of large-scale paired datasets for scene sketches. In this paper, we propose a self-supervised method for scene sketch generation that does not rely on any existing scene sketch, enabling the transformation of single-object sketches into scene sketches. To accomplish this, we introduce a method for vector sketch captioning and sketch semantic expansion. Additionally, we design a sketch generation network that incorporates a fusion of multi-modal perceptual constraints, suitable for application in zero-shot image-to-sketch downstream task, demonstrating state-of-the-art performance through experimental validation. Finally, leveraging our proposed sketch-to-sketch generation method, we contribute a large-scale dataset centered around scene sketches, comprising highly semantically consistent "text-sketch-image" triplets. Our research can provide critical technical support for AI content creation and customized data generation based on IoT devices.
BibTeX
@article{wu2025sketchtriplet,
  author = {Zhenbei Wu and Qiang Wang and Jie Yang},
  title = {{SketchTriplet: Self-Supervised Scenarized Sketch–Text–Image Triplet Generation}},
  journal = {IEEE Internet of Things Journal},
  year = {2025},
  doi = {10.1109/jiot.2024.3523382},
}

Loading related content...

Figure