Sketch-guided Image Inpainting with Partial Discrete Diffusion Process

Nakul Sharma, Aditay Tripathi, Anirban Chakraborty, Anand MishraWorkshops @ CVPR 2024 · DOI · arXiv

Introduction

Image inpainting—the process of reconstructing missing or masked regions in a photograph—has seen significant advancements through generative modeling. While modern tools effectively leverage surrounding context or text prompts to fill gaps, they often lack the fine-grained control required for specific object poses or structural details. Text descriptions like "add a dog here" frequently fail to capture the precise geometry a user might envision.

Sketching offers an intuitive modality for communicating structure, yet it introduces a substantial "domain gap": simple black-and-white line drawings bear little resemblance to high-resolution natural images. Previous sketch-guided methods often struggled to integrate complete, realistic objects into complex scenes. This paper introduces a novel framework centered on a Partial Discrete Diffusion Process (PDDP) to bridge this gap, enabling precise, sketch-driven object-level inpainting.

Methodology: Partial Discrete Diffusion

The proposed method operates in a two-stage framework that treats inpainting as a conditional generation task within a discrete latent space.

Discrete Latent Space and PDDP

Rather than operating on raw pixels, the model utilizes a Vector Quantized Variational Autoencoder (VQ-VAE) to represent images as sequences of discrete tokens from a learned codebook. This discretization simplifies visual data into a format amenable to transformer-based modeling.

The core innovation is the Partial Discrete Diffusion Process (PDDP). Unlike standard diffusion models that corrupt an entire image into noise, PDDP only applies noise to the tokens within the masked (missing) region. This ensures that the unmasked portion of the image remains intact, providing a stable context for the generation process.

The architecture of the Sketch-guided Bi-directional Transformer and the Partial Discrete Diffusion Process.

Sketch-Guided Bi-directional Transformer

As shown in the architecture diagram above, the "brain" of the system is a bi-directional transformer. It processes two primary inputs: the context tokens from the unmasked image and the query sketch. The sketch is encoded via a ResNet50 backbone and integrated with 2D positional embeddings. The transformer then predicts the original, noise-free tokens for the masked area by attending to both the structural guidance of the sketch and the surrounding visual textures simultaneously. This iterative refinement allows the model to "translate" geometric lines into photorealistic textures that match the lighting and perspective of the original scene.

Dataset and Training

To train the model, the authors curated a large-scale dataset derived from MS-COCO. Objects were segmented and enhanced using ESR-GAN for high-quality detail. These objects were then converted into realistic, hand-drawn-style sketches using the PhotoSketching tool. The resulting dataset comprises 860,000 training pairs, providing the model with a vast library of object-sketch correspondences.

Results and Evaluation

The proposed method was evaluated against state-of-the-art baselines, including ControlNet and Palette.

Quantitative Performance

On the Fréchet Inception Distance (FID) metric—a standard measure of image realism and diversity—the model achieved a score of 7.72, significantly outperforming ControlNet's 10.77. Furthermore, in user studies, the model's results were preferred 68.54% of the time. It also demonstrated superior structural fidelity, receiving a consistency score of 4.34/5 for how accurately the output followed the input sketch.

Qualitative Comparison

As illustrated in the comparison below, the PDDP approach maintains significantly higher fidelity to the original sketch while producing more coherent textures than existing GAN-based or diffusion-based methods.

Qualitative comparison of the proposed method against DeFLOCNet, DeepFillv2, Palette, and ControlNet across various categories.

Conclusion and Future Work

By combining discrete diffusion with bi-directional transformer modeling, this research provides a robust solution for sketch-guided image editing. The PDDP framework effectively handles the domain gap between sketches and photos, offering users unprecedented control over inpainting results. Future work aims to extend this capability to partial or sparse sketches and to refine the integration of sketch features with highly complex background textures.

Summary updated

BibTeX
@inproceedings{sharma2024sketch,
  author = {Nakul Sharma and Aditay Tripathi and Anirban Chakraborty and Anand Mishra},
  title = {{Sketch-guided Image Inpainting with Partial Discrete Diffusion Process}},
  booktitle = {Workshops @ CVPR},
  year = {2024},
  doi = {10.1109/cvprw63382.2024.00609},
}
Figure