Inharmonious Region Localization

Jing Liang, Li Niu, Liqing Zhang

Abstract

Image editing often introduces "inharmonious" regions where the lighting or color of a pasted object fails to match the background. This paper introduces the Deep Inharmonious Region Localization (DIRL) network, which utilizes novel bi-directional feature integration and mask-guided attention to accurately pinpoint these manipulated areas. By effectively fusing multi-scale context while explicitly suppressing redundant background noise, DIRL significantly outperforms existing image manipulation and segmentation models.

Introduction

With modern image editing tools, it is incredibly easy to copy and paste objects into new backgrounds. However, these synthetic images often look slightly "off" because the color and lighting statistics of the manipulated region clash with the background. While image harmonization models exist to fix these mismatches, they typically require a manually provided mask of the edited region. Conversely, existing image manipulation localization models focus on detecting digital artifacts (like JPEG compression or camera noise) rather than visual color/lighting inconsistencies.

To automatically localize these inharmonious regions, a neural network must compare a specific area against its multi-scale surroundings. The core challenge is that fusing multi-scale contextual information inevitably introduces redundant, distracting data. This research addresses how to thoroughly fuse multi-scale features to spot lighting/color inconsistencies while aggressively filtering out noise.

Proposed Method: DIRL

The authors propose the Deep Inharmonious Region Localization (DIRL) network, built on a UNet-like encoder-decoder architecture using a ResNet34 backbone. The architecture introduces three novel blocks to handle feature fusion and redundancy:

The DIRL architecture featuring Bi-directional Feature Integration (BFI), Mask-guided Dual Attention (MDA), and Global-context Guided Decoder (GGD) blocks.
  1. Bi-directional Feature Integration (BFI) Block: Placed between the encoder and decoder, BFI fuses adjacent multi-scale features from the encoder. Unlike previous methods that simply merge neighboring features, BFI uses parallel top-down and bottom-up streams, allowing high-level and low-level features to exchange information bi-directionally for a more holistic understanding of image context.
  2. Mask-guided Dual Attention (MDA) Block: To counter the distracting, redundant information introduced by multi-scale fusion, the MDA block applies spatial and channel attention to the BFI outputs. Crucially, the spatial attention map is directly supervised by the ground-truth mask, forcing the network to focus strictly on the inharmonious region and preventing "attention drift" toward complex background elements.
  3. Global-context Guided Decoder (GGD) Block: Global context (e.g., overall scene type or lighting) is vital for determining if a local region is out of place. The GGD block injects this high-level global context directly into every level of the decoder via a shortcut connection, guiding the final mask prediction.

Experiments and Results

The authors evaluated DIRL using a tailored version of the iHarmony4 dataset. To avoid ambiguity in cases where a manipulated foreground takes up more than 50% of the image, they filtered the dataset to remove such instances, resulting in 64,255 training images and 7,237 test images.

Quantitative Performance

DIRL achieves state-of-the-art performance, significantly beating general segmentation models, image manipulation localization models, and image harmonization models. As shown in the table below, DIRL achieved 80.02% Average Precision (AP), a 0.7317 F1 score, and 67.85% Intersection over Union (IoU).

Quantitative comparison of DIRL against various baseline models on the tailored iHarmony4 dataset.

DIRL outperformed the strongest baseline (DeepLabv3) by +4.33% in AP and +4.15% in F1. It also significantly exceeded domain-specific models like MAGritte and S2AM.

Qualitative Insights

Visualizations demonstrate that DIRL produces masks with much sharper boundaries and fewer artifacts compared to baselines. It successfully identifies intact inharmonious regions even in highly cluttered scenes where other models fail. Key insights from the study include:

  • Multi-scale fusion is a double-edged sword: Explicit redundancy suppression (via the MDA block) is as important as the fusion itself.
  • Mask supervision prevents "attention drift": Explicitly penalizing attention maps when they deviate from the ground-truth mask helps the network ignore distractors.
  • Global context needs to be ubiquitous: Injecting global lighting/scene information into every decoder layer provides a stronger guiding signal than natural trickle-down propagation.

Conclusion and Future Work

DIRL effectively addresses the challenge of localizing inharmonious regions by balancing multi-scale feature integration with targeted noise suppression. A primary limitation remains the ">50% ambiguity problem," where the model struggles to define the anomaly if the foreground dominates the image. Future work involves integrating this localization network into end-to-end automatic image harmonization pipelines for fully automated detection and correction of image composites.

Summary updated

BibTeX
@inproceedings{liang2021inharmonious,
  author = {Jing Liang and Li Niu and Liqing Zhang},
  title = {{Inharmonious Region Localization}},
  booktitle = {ICME},
  year = {2021},
  doi = {10.1109/icme51207.2021.9428309},
}
Figure