Introduction
Image inpainting—the process of reconstructing missing or masked regions in a photograph—has seen significant advancements through generative modeling. While modern tools effectively leverage surrounding context or text prompts to fill gaps, they often lack the fine-grained control required for specific object poses or structural details. Text descriptions like "add a dog here" frequently fail to capture the precise geometry a user might envision.
Sketching offers an intuitive modality for communicating structure, yet it introduces a substantial "domain gap": simple black-and-white line drawings bear little resemblance to high-resolution natural images. Previous sketch-guided methods often struggled to integrate complete, realistic objects into complex scenes. This paper introduces a novel framework centered on a Partial Discrete Diffusion Process (PDDP) to bridge this gap, enabling precise, sketch-driven object-level inpainting.
Methodology: Partial Discrete Diffusion
The proposed method operates in a two-stage framework that treats inpainting as a conditional generation task within a discrete latent space.
Discrete Latent Space and PDDP
Rather than operating on raw pixels, the model utilizes a Vector Quantized Variational Autoencoder (VQ-VAE) to represent images as sequences of discrete tokens from a learned codebook. This discretization simplifies visual data into a format amenable to transformer-based modeling.
The core innovation is the Partial Discrete Diffusion Process (PDDP). Unlike standard diffusion models that corrupt an entire image into noise, PDDP only applies noise to the tokens within the masked (missing) region. This ensures that the unmasked portion of the image remains intact, providing a stable context for the generation process.
Sketch-Guided Bi-directional Transformer
As shown in the architecture diagram above, the "brain" of the system is a bi-directional transformer. It processes two primary inputs: the context tokens from the unmasked image and the query sketch. The sketch is encoded via a ResNet50 backbone and integrated with 2D positional embeddings. The transformer then predicts the original, noise-free tokens for the masked area by attending to both the structural guidance of the sketch and the surrounding visual textures simultaneously. This iterative refinement allows the model to "translate" geometric lines into photorealistic textures that match the lighting and perspective of the original scene.
Dataset and Training
To train the model, the authors curated a large-scale dataset derived from MS-COCO. Objects were segmented and enhanced using ESR-GAN for high-quality detail. These objects were then converted into realistic, hand-drawn-style sketches using the PhotoSketching tool. The resulting dataset comprises 860,000 training pairs, providing the model with a vast library of object-sketch correspondences.
Results and Evaluation
The proposed method was evaluated against state-of-the-art baselines, including ControlNet and Palette.
Quantitative Performance
On the Fréchet Inception Distance (FID) metric—a standard measure of image realism and diversity—the model achieved a score of 7.72, significantly outperforming ControlNet's 10.77. Furthermore, in user studies, the model's results were preferred 68.54% of the time. It also demonstrated superior structural fidelity, receiving a consistency score of 4.34/5 for how accurately the output followed the input sketch.
Qualitative Comparison
As illustrated in the comparison below, the PDDP approach maintains significantly higher fidelity to the original sketch while producing more coherent textures than existing GAN-based or diffusion-based methods.
Conclusion and Future Work
By combining discrete diffusion with bi-directional transformer modeling, this research provides a robust solution for sketch-guided image editing. The PDDP framework effectively handles the domain gap between sketches and photos, offering users unprecedented control over inpainting results. Future work aims to extend this capability to partial or sparse sketches and to refine the integration of sketch features with highly complex background textures.
Summary updated