Context-Aware Image Completion

Jinoh Cho, Minguk Kang, Vibhav Vineet, Jaesik Park

Abstract

Image completion is a task that aims to fill in the missing region of a masked image with plausible contents. However, existing image completion methods tend to fill in the missing region with the surrounding texture instead of hallucinating a visual instance that is suitable in accordance with the context of the scene. In this work, we propose a novel image completion model, dubbed ImComplete, that hallucinates the missing instance that harmonizes well with - and thus preserves - the original context. ImComplete first adopts a transformer architecture that considers the visible instances and the location of the missing region. Then, ImComplete completes the semantic segmentation masks within the missing region, providing pixel-level semantic and structural guidance. Finally, the image synthesis blocks generate photo-realistic content. We perform a comprehensive evaluation of the results in terms of visual quality (LPIPS and FID) and contextual preservation scores (CLIPscore and object detection accuracy) with COCO-panoptic and Visual Genome datasets. Experimental results show the superiority of ImComplete on various natural images.

Introduction

Standard image inpainting often fails by simply "smearing" surrounding textures into a hole, losing the original context of the scene. While GANs and Diffusion models have improved visual fidelity, they often exhibit a "texture-filling" bias. For instance, masking a person on a bench might result in a model extending the bench and grass, erasing the semantic context of the scene. The authors argue that true image completion should be instance-aware, hallucinating specific objects (like a surfboard near a surfer) implied by the surrounding context.

The ImComplete Framework

The proposed ImComplete model utilizes a three-stage pipeline to maintain scene logic by separating the "what" (class), the "where/shape" (mask), and the "how it looks" (pixels).

The three-stage ImComplete pipeline: (1) Missing Instance Inference, (2) Segmentation Completion, and (3) Segmentation-Guided Image Completion.
  1. Missing Instance Prediction: A pre-trained DETR (Detection Transformer) identifies visible objects. These tokens (class + bounding box) are processed by a specialized transformer to predict the category of the missing instance based on object co-occurrence.
  2. Semantic Segmentation Map Generation: The model generates a plausible semantic mask for the predicted class. This involves an Instance Segmentation Generator for the object and a Background Segmentation Completion Network to fill the remaining mask area.
  3. Segmentation-Guided Image Completion: The completed semantic map serves as a blueprint for "plug-and-play" synthesis blocks (e.g., SPADE, OASIS, or Stable Diffusion) to generate final pixels.

Experimental Results

The model was evaluated on COCO-panoptic and Visual Genome datasets. ImComplete significantly outperforms state-of-the-art baselines like MAT and RePaint in maintaining scene context.

Qualitative comparison of ImComplete against state-of-the-art baselines. Note how ImComplete restores semantically appropriate objects (donuts, fire hydrant, pitcher) guided by the restored segmentation.

Quantitative Performance

  • Contextual Accuracy: On COCO, ImComplete achieved a DETR Accuracy of 40.22%, vastly higher than MAT (1.89%) and RePaint (9.35%), proving its ability to restore the correct object type.
  • Semantic Alignment: It achieved a leading CLIPScore of 0.663, indicating high fidelity to the original scene's semantic description.
  • Visual Quality: The model maintained competitive FID (6.714) and LPIPS scores, ensuring generated instances are photo-realistic.

Conclusion and Future Work

ImComplete demonstrates that treating inpainting as a logical puzzle—using intermediate semantic masks—prevents artifacts like "melting" and preserves scene meaning. Future work includes expanding the model to an "open-set" vocabulary beyond the current 30 COCO classes and integrating advanced text-to-image techniques for greater diversity and user control.

Summary updated

Abstract

Image completion is a task that aims to fill in the missing region of a masked image with plausible contents. However, existing image completion methods tend to fill in the missing region with the surrounding texture instead of hallucinating a visual instance that is suitable in accordance with the context of the scene. In this work, we propose a novel image completion model, dubbed ImComplete, that hallucinates the missing instance that harmonizes well with - and thus preserves - the original context. ImComplete first adopts a transformer architecture that considers the visible instances and the location of the missing region. Then, ImComplete completes the semantic segmentation masks within the missing region, providing pixel-level semantic and structural guidance. Finally, the image synthesis blocks generate photo-realistic content. We perform a comprehensive evaluation of the results in terms of visual quality (LPIPS and FID) and contextual preservation scores (CLIPscore and object detection accuracy) with COCO-panoptic and Visual Genome datasets. Experimental results show the superiority of ImComplete on various natural images.
BibTeX
@inproceedings{cho2023context,
  author = {Jinoh Cho and Minguk Kang and Vibhav Vineet and Jaesik Park},
  title = {{Context-Aware Image Completion}},
  booktitle = {International Conference on Learning Representations},
  year = {2023},
  url = {https://openreview.net/forum?id=YlmzborbHTy},
}
Figure