Abstract
Image completion is a task that aims to fill in the missing region of a masked image with plausible contents. However, existing image completion methods tend to fill in the missing region with the surrounding texture instead of hallucinating a visual instance that is suitable in accordance with the context of the scene. In this work, we propose a novel image completion model, dubbed ImComplete, that hallucinates the missing instance that harmonizes well with - and thus preserves - the original context. ImComplete first adopts a transformer architecture that considers the visible instances and the location of the missing region. Then, ImComplete completes the semantic segmentation masks within the missing region, providing pixel-level semantic and structural guidance. Finally, the image synthesis blocks generate photo-realistic content. We perform a comprehensive evaluation of the results in terms of visual quality (LPIPS and FID) and contextual preservation scores (CLIPscore and object detection accuracy) with COCO-panoptic and Visual Genome datasets. Experimental results show the superiority of ImComplete on various natural images.
Introduction
Standard image inpainting often fails by simply "smearing" surrounding textures into a hole, losing the original context of the scene. While GANs and Diffusion models have improved visual fidelity, they often exhibit a "texture-filling" bias. For instance, masking a person on a bench might result in a model extending the bench and grass, erasing the semantic context of the scene. The authors argue that true image completion should be instance-aware, hallucinating specific objects (like a surfboard near a surfer) implied by the surrounding context.
The ImComplete Framework
The proposed ImComplete model utilizes a three-stage pipeline to maintain scene logic by separating the "what" (class), the "where/shape" (mask), and the "how it looks" (pixels).
- Missing Instance Prediction: A pre-trained DETR (Detection Transformer) identifies visible objects. These tokens (class + bounding box) are processed by a specialized transformer to predict the category of the missing instance based on object co-occurrence.
- Semantic Segmentation Map Generation: The model generates a plausible semantic mask for the predicted class. This involves an Instance Segmentation Generator for the object and a Background Segmentation Completion Network to fill the remaining mask area.
- Segmentation-Guided Image Completion: The completed semantic map serves as a blueprint for "plug-and-play" synthesis blocks (e.g., SPADE, OASIS, or Stable Diffusion) to generate final pixels.
Experimental Results
The model was evaluated on COCO-panoptic and Visual Genome datasets. ImComplete significantly outperforms state-of-the-art baselines like MAT and RePaint in maintaining scene context.
Quantitative Performance
- Contextual Accuracy: On COCO, ImComplete achieved a DETR Accuracy of 40.22%, vastly higher than MAT (1.89%) and RePaint (9.35%), proving its ability to restore the correct object type.
- Semantic Alignment: It achieved a leading CLIPScore of 0.663, indicating high fidelity to the original scene's semantic description.
- Visual Quality: The model maintained competitive FID (6.714) and LPIPS scores, ensuring generated instances are photo-realistic.
Conclusion and Future Work
ImComplete demonstrates that treating inpainting as a logical puzzle—using intermediate semantic masks—prevents artifacts like "melting" and preserves scene meaning. Future work includes expanding the model to an "open-set" vocabulary beyond the current 30 COCO classes and integrating advanced text-to-image techniques for greater diversity and user control.
Summary updated