Context-Aware Image Completion

Jinoh Cho, Minguk Kang, Vibhav Vineet, Jaesik Park

TL;DR

When a focal object is completely erased from a photo, standard image inpainting models usually smear the surrounding background across the hole rather than hallucinating a missing object. Refill is a three-stage image completion framework that reasons about visible scene context to predict what object is missing, reconstructs an explicit semantic segmentation mask, and uses that mask to synthesize a contextually appropriate, photo-realistic object with seamless boundaries.

What Problem Does This Paper Address?

Traditional image completion models (like MAT or EdgeConnect) excel at texture synthesis and local edge completion. However, when an entire object disappears—such as a horse beneath a riding person—these models simply extend the sand or grass, radically shifting the scene’s semantic meaning (e.g., turning a horseback rider into an awkward hovering person or a standing pedestrian).

Prior work attempting full object hallucination (HVITA) suffered from three major drawbacks:

  1. It required rectangular bounding box holes, failing to generalize to irregular masks.
  2. It pasted generated patches directly into the scene, causing visible seam artifacts.
  3. It relied on a fragile post-processing refinement network.

Refill seeks to restore semantic context for arbitrary hole shapes without requiring cut-and-paste patch insertion or separate refinement stages.

Methods

Refill divides context-aware inpainting into three decoupled steps:

[Masked Image] 
       │
       ▼ (DETR feature extraction)
[Step 1: Missing Instance Prediction] ──> Predicts missing class label (Transformer)
       │
       ▼
[Step 2: Dual Segmentation Generation]
       ├── Foreground Mask: BigGAN generator conditioned on predicted class & bbox
       └── Background Mask: Transformer-body completion net repairing scene context
       │
       ▼ (Fused Panoptic Mask)
[Step 3: Guided Image Completion] ──> SPADE/OASIS-conditioned UNet generates RGB pixels
  1. Missing Instance Inference (Transformer): Visible objects and the missing hole are detected using a pre-trained DETR model. A 12-layer Transformer encoder takes the visible classes, their normalized bounding-box coordinates (using 4-coordinate absolute positional encodings: $C_x, C_y, W, H$), and a masked-region token to predict the missing object's category.
  2. Intermediate Semantic Mask Synthesis: Instead of synthesizing pixels directly, the model splits the hole into foreground and background semantic masks:
  3. Foreground: Generated by a modified BigGAN conditioned on the predicted class label and bounding box.
  4. Background: Repaired using an 8-layer Transformer-body completion network that outperforms standard CNNs on large masks.
  5. The two masks are composited into a single complete panoptic layout.
  6. Segmentation-Guided Image Completion: A UNet conditioned with SPADE or OASIS normalization layers translates the synthesized segmentation layout into final RGB pixels, ensuring structural consistency with the unmasked boundaries.

Datasets

  • COCO-panoptic: Used for training and validation (118K training, 5K validation images). Missing regions were created by picking bounding boxes covering 30 frequent "things" classes, expanded into irregular brush strokes covering 10% to 50% of the image area.
  • Visual Genome: Used to evaluate zero-shot transfer (110K images with 34K categories mapped to the COCO class space via DETR predictions).

Key Insights and Ideas

  • Decoupling Semantics from Texture: End-to-end models struggle to simultaneously guess missing scene semantics and synthesize high-frequency pixels. Breaking the task into classification $\to$ segmentation $\to$ neural rendering makes each step tractable and modular.
  • Explicit Geometry Matters for Transformers: The ablation study showed that feeding bounding box geometry ($C_x, C_y, W, H$) via linear projections outperformed relative coordinates, learnable embeddings, or graph convolutional networks (GCNs) when inferring the missing class.
  • Transformers Excel at Mask Inpainting: While CNN backbones introduce blur and repetitive artifacts when filling large segmentation holes, a Transformer-based background mask completion module retained structural coherence across holes covering >70% of the mask.
  • New Semantic Evaluation Metrics: Traditional pixel-level metrics like LPIPS penalize diverse, valid completions. The authors introduced CLIPScore (text-image alignment to an automated caption of the missing content) and Visual Grounding Accuracy (VGA) (whether a visual grounding model can locate the expected object in the completed image).

Key Results

Metric MAT (Baseline) HVITA (Baseline) Refill (OASIS)
COCO VGA (%) $\uparrow$ 3.32% 17.87% 30.35%
COCO CLIPScore $\uparrow$ 0.614 0.567 0.641
COCO FID $\downarrow$ 7.192 10.496 7.284
VG (Zero-shot) FID $\downarrow$ 6.488 9.311 5.849
VG (Zero-shot) VGA (%) $\uparrow$ 3.24% 18.61% 32.71%
  • Preserving Context: Refill achieved a 30.35% visual grounding accuracy on COCO—nearly double HVITA (17.87%) and an order of magnitude higher than MAT (3.32%), proving that standard baselines rarely restore missing entities.
  • Visual Realism: Refill reached an FID of 7.284 on COCO (closely matching MAT's 7.192, despite MAT having been pre-trained on the 8-million-image Places365 dataset) and achieved the best FID (5.849) on zero-shot Visual Genome.
  • Class Prediction Accuracy: The missing instance transformer achieved 67.55% accuracy on COCO (vs. 50.66% for HVITA’s GCN).

Limitations and Future Work

  • Dependence on Semantic Synthesis Backbones: Generation diversity and fine details are constrained by the underlying SPADE/OASIS decoders.
  • Absence of a "No Instance" Hypothesis: The pipeline always attempts to insert an object even when the scene context would naturally call for pure background.
  • High-Order Spatial Reasoning: The model lacks explicit reasoning for occlusion order, correct perspective scale, and 3D instance orientation.
  • Closed Vocabulary: The current system is restricted to 30 predefined COCO categories; integrating open-vocabulary segmentation and modern text-to-image diffusion priors is an obvious next frontier.

Summary updated

Abstract

Image completion is a task that aims to fill in the missing region of a masked image with plausible contents. However, existing image completion methods tend to fill in the missing region with the surrounding texture instead of hallucinating a visual instance that is suitable in accordance with the context of the scene. In this work, we propose a novel image completion model, dubbed Refill, that hallucinates the missing instance that harmonizes well with - and thus preserves - the original context. Refill first adopts a transformer architecture that considers the types, locations of the visible instances, and the location of the missing region. Then, Refill completes the missing foreground and background semantic segmentation masks within the missing region, providing pixel-level semantic and structural guidance to generate missing contents with seamless boundaries. Finally, we condition the image synthesis blocks by using the completed segmentation mask to generate photo-realistic contents to fill out the missing region. Experimental results show the superiority of Refill over state-of-the-art image completion approaches on various natural images.
BibTeX
@inproceedings{cho2023context,
  author = {Jinoh Cho and Minguk Kang and Vibhav Vineet and Jaesik Park},
  title = {{Context-Aware Image Completion}},
  booktitle = {International Conference on Learning Representations},
  year = {2023},
  url = {https://openreview.net/forum?id=YlmzborbHTy},
}

Loading related content...

Figure