UniCoRN: A Unified Conditional Image Repainting Network

Jimeng Sun, Shuchen Weng, Zheng Chang, Si Li, Boxin Shi

Abstract

Conditional Image Repainting (CIR) involves filling a masked region of an image with new content guided by multi-modal inputs such as color, texture, and geometry. Existing state-of-the-art methods typically employ a two-phase "generate then blend" strategy, which frequently leads to color-image incongruity and boundary artifacts. We introduce UniCoRN (Unified Conditional Image Repainting Network), a framework that unifies these phases by incorporating background information as a primary constraint during the generation process. Our approach utilizes a Cross-Modality Condition Fusion Module (CMCFM) and a Hierarchical Cross-Modality Similarity Model (HCMSM) to ensure global and local consistency. Experimental results on the PERSON-CIR and a new LANDSCAPE-CIR dataset demonstrate that UniCoRN significantly outperforms existing methods in both quantitative metrics and human perception.

The Challenge of Two-Phase Repainting

The primary limitation in current CIR research is the reliance on a two-phase pipeline. In the traditional approach, a generator first creates the target object based on user conditions in isolation from the surrounding background. Subsequently, a separate compositing or harmonization model attempts to blend this new object into the original scene.

Comparison between the proposed Unified CIR (UniCoRN) and the traditional Two-phase CIR approach.

As illustrated in the comparison above, this disjointed process causes two major issues:

  1. Color-Image Incongruity: Because the generator is unaware of the background's lighting and tone during the initial synthesis, the resulting object often appears as a misplaced cutout.
  2. Gradient Truncation: The transition between independent phases can truncate gradients during training, leading to visible artifacts at the mask boundaries where textures fail to align naturally.

The UniCoRN Architecture

UniCoRN eliminates the need for a separate compositing step by treating the background as a fundamental prior. The network is built upon three core components:

Cross-Modality Condition Fusion Module (CMCFM)

The CMCFM acts as a spatial mixer operating in a high-dimensional feature space. It uses the geometry mask as a gating mechanism to fuse background features with the features of the region to be repainted. This ensures that the generated content is environmentally aware from the earliest stages of synthesis.

Feature Adaptive Batch Normalization (FABN)

Modified from the SPADE/GauGAN architecture, FABN injects the fused multi-modal features—encompassing color, texture, and background—directly into the generator's layers. This allows the model to adaptively modulate the synthesis process based on the complex interplay of input conditions.

Hierarchical Cross-Modality Similarity Model (HCMSM)

To enforce strict adherence to color and texture descriptions, the HCMSM acts as a multi-scale supervisor. By using a pyramid of encoders, it checks for consistency at various semantic levels. This prevents the "global blur" problem, ensuring that color attributes are applied accurately both at a macro level (e.g., the overall color of a garment) and a micro level (e.g., shadows and highlights within fabric folds).

Experimental Evaluation

The researchers evaluated UniCoRN against two benchmarks: PERSON-CIR, containing 48K images of people with detailed parsing masks, and LANDSCAPE-CIR, a new dataset comprising 31,000 high-resolution outdoor scenes with automated masks and color attributes.

Quantitative and Qualitative Results

UniCoRN achieved a state-of-the-art Frechet Inception Distance (FID) of 11.45 on the PERSON-CIR dataset, a substantial improvement over the previous best of 13.74. It also demonstrated superior R-precision, indicating a higher correlation between the generated images and the input color descriptions.

Qualitatively, UniCoRN produces images with significantly smoother boundaries and more realistic textures compared to two-phase models like MISC and SEAN. In a crowd-sourced user study on Amazon Mechanical Turk, UniCoRN's results were preferred over 50% of the time when compared against four leading competitors.

Conclusion and Future Work

UniCoRN represents a paradigm shift in image editing by moving from a sequential "pasting" logic to a unified synthesis framework. By integrating the background as a primary condition, the model achieves unprecedented visual harmony. While the current model is computationally intensive with 103.1 million parameters, future research will focus on knowledge distillation to create lightweight versions suitable for real-time mobile applications.

Summary updated

Abstract

Conditional image repainting (CIR) is an advanced image editing task, which requires the model to generate visual content in user-specified regions conditioned on multiple cross-modality constraints, and composite the visual content with the provided background seamlessly. Existing methods based on two-phase architecture design assume dependency between phases and cause color-image incongruity. To solve these problems, we propose a novel Unified Conditional image Repainting Network (UniCoRN). We break the two-phase assumption in the CIR task by constructing the interaction and dependency relationship between background and other conditions. We further introduce the hierarchical structure into cross-modality similarity model to capture feature patterns at different levels and bridge the gap between visual content and color condition. A new LANDSCAPE-CIR dataset is collected and annotated to expand the application scenarios of the CIR task. Experiments show that UniCoRN achieves higher synthetic quality, better condition consistency, and more realistic compositing effect.
BibTeX
@inproceedings{sun2022unicorn,
  author = {Jimeng Sun and Shuchen Weng and Zheng Chang and Si Li and Boxin Shi},
  title = {{UniCoRN: A Unified Conditional Image Repainting Network}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2022},
  doi = {10.1109/cvpr52688.2022.01108},
}
Figure