Image Harmonization with Transformer

Zonghui Guo, Dongsheng Guo, Haiyong Zheng, Zhaorui Gu, Bing Zheng, Junyu DongICCV 2021 · DOI

Abstract

Image harmonization is the process of adjusting the appearance of a foreground object to ensure it blends naturally with a new background. While Convolutional Neural Networks (CNNs) have been the standard for this task, they often fail to capture the global context necessary for realistic lighting alignment. This paper presents the first Transformer-based framework for image harmonization. By leveraging self-attention mechanisms and a disentangled architecture that separates material properties from illumination, the proposed model achieves state-of-the-art performance on the iHarmony4 benchmark and demonstrates superior realism in human subjective tests.

Introduction

When compositing an object from one image into another, the resulting "copy-paste" effect often looks artificial due to discrepancies in lighting, color temperature, and shadows. Image harmonization seeks to resolve these inconsistencies. Traditional CNN-based approaches are inherently limited by their local receptive fields, making it difficult to relate a foreground object to distant light sources or global scene characteristics. To address this, we propose using Transformers to capture long-range dependencies, ensuring that the foreground is harmonized based on the entire background context.

Methodology

The proposed framework introduces two primary architectures designed to overcome the limitations of local feature extraction:

Harmonization Transformer (HT)

The HT architecture utilizes a CNN encoder to extract initial features, which are then processed by a Transformer Encoder (TRE). The self-attention mechanism within the TRE allows every pixel in the foreground to attend to every pixel in the background, facilitating a global understanding of the scene's illumination.

Disentangled Harmonization Transformer (D-HT)

Building on Retinex Theory—which posits that an image is the product of reflectance (material) and illumination (light)—the D-HT employs a dual-pathway system.

  • Reflectance Path: Preserves the material properties, such as shape and texture, of the foreground object.
  • Illumination Path: Extracts light information from the background. The model then re-renders the foreground by applying the extracted background illumination to the foreground's reflectance, resulting in a more physically plausible composite that avoids "smearing" textures while fixing colors.

Experiments and Results

The models were evaluated on the iHarmony4 dataset, the industry-standard benchmark consisting of four sub-datasets: HCOCO, HAdobe5k, HFlickr, and Hday2night.

Quantitative Performance

The D-HT model achieved a foreground Mean Squared Error (fMSE) of 320.78, significantly outperforming the previous state-of-the-art model, DoveNet, which recorded an fMSE of 541.53.

Qualitative Comparison

Visual results demonstrate the model's ability to handle complex lighting scenarios, such as moving an object from a sunny outdoor scene to a dimly lit indoor party. As shown in the comparison below, the proposed method produces much more realistic results compared to naive compositing and previous CNN-based methods.

Qualitative comparison of image harmonization results. The proposed method ("Ours") shows superior blending of foreground objects into diverse backgrounds compared to DoveNet and simple compositing.

Human Evaluation and Versatility

In a subjective study involving 60 participants, the proposed method received a Bradley-Terry (B-T) score of 2.248, more than double that of the nearest competitor (1.032), confirming a strong human preference for the realism of our results. Furthermore, the D-HT architecture proved versatile, performing excellently in secondary tasks like image inpainting and low-light enhancement.

Conclusion

By shifting from local CNNs to global Transformers, this work sets a new standard for image harmonization. The disentangled representation of light and material allows for high-fidelity results that respect both the object's identity and the scene's environment. Future work will explore more sophisticated light representations to further enhance physical accuracy.

Summary updated

BibTeX
@inproceedings{guo2021image,
  author = {Zonghui Guo and Dongsheng Guo and Haiyong Zheng and Zhaorui Gu and Bing Zheng and Junyu Dong},
  title = {{Image Harmonization with Transformer}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2021},
  doi = {10.1109/iccv48922.2021.01460},
}
Figure