Abstract
Image inpainting is the task of plausibly restoring missing pixels within a hole region that is to be removed from a target image. Most existing technologies exploit patch similarities within the image, or leverage large-scale training data to fill the hole using learned semantic and texture information. However, due to the ill-posed nature of the inpainting task, such methods struggle to complete larger holes containing complicated scenes. In this paper, we propose TransFill, a multi-homography transformed fusion method to fill the hole by referring to another source image that shares scene contents with the target image. We first align the source image to the target image by estimating multiple homographies guided by different depth levels. We then learn to adjust the color and apply a pixel-level warping to each homography-warped source image to make it more consistent with the target. Finally, a pixel-level fusion module is learned to selectively merge the different proposals. Our method achieves state-of-the-art performance on pairs of images across a variety of wide baselines and color differences, and generalizes to user-provided image pairs.
Introduction
Image inpainting—the art of removing unwanted objects and filling the resulting "hole"—is notoriously difficult when the missing region is large or contains complex structures. While single-image methods (which look at the rest of the photo for clues) have improved with deep learning, they often "hallucinate" blurry or semantically incorrect textures.
A better approach is to use a reference image (e.g., another photo of the same scene taken from a slightly different angle). However, this introduces three major technical hurdles:
- Parallax: Because the camera moved, objects at different depths shift by different amounts, meaning a simple 2D stretch (homography) won't align the images.
- Photometric Inconsistency: Differences in exposure, white balance, or lighting between the two shots make "copy-pasting" look obvious.
- Occlusions: Some parts of the hole might not be visible in the reference image at all.
Methodology
The authors propose a four-stage pipeline called TransFill:
- Multi-homography Proposals: Instead of trying to find one perfect alignment, the system uses monocular depth estimation to cluster the scene into different depth layers. It then generates multiple "proposals," each aligned to a specific depth plane.
- Color-Spatial Transformer (CST): Each proposal undergoes a two-part refinement. First, it uses a learned bilateral grid to adjust colors and exposure while preserving edge details. Second, it applies a per-pixel spatial warp to fix minor misalignments that the initial homography missed.
- Single-Proposal Fusion (SPF): Each refined proposal is compared against a "backup" version created by a state-of-the-art single-image inpainter (ProFill). This helps the model learn a confidence map—essentially deciding which parts of the reference image are trustworthy.
- Multi-Proposal Fusion (MPF): Finally, a UNet-based module merges all the depth-specific proposals and the single-image inpainting result into one seamless final image.
Datasets and Evaluation
To train and validate the model, the researchers utilized several key datasets:
- RealEstate10K: A large-scale dataset of YouTube walk-through videos, providing frame pairs with varying displacements to simulate camera movement and parallax.
- MIT-Adobe5K: Used to augment training data with professional color and exposure variations.
- User-Provided Pairs: A collection of 3,000 real-world image pairs used to test generalization to unconstrained photography.
Key Insights and Results
TransFill significantly outperformed existing baselines (like OPN and DFG) across both quantitative metrics and human perception. On the RealEstate10K dataset, TransFill achieved a PSNR of 37.58, compared to 32.43 for the nearest reference-based competitor. In a blind user study, participants preferred TransFill results between 81% and 95% of the time.
Key Innovations: * Depth-Aware Alignment: Acknowledging that a single global transformation is insufficient for 3D scenes allows the model to accurately map both foreground and background. * Decoupling Color and Geometry: Using bilateral grids for color and separate warping fields for geometry avoids "ghosting" artifacts. * Hybrid Safety Net: By integrating single-image inpainting (ProFill), the model can gracefully fill regions that are occluded in the reference photo.
Limitations and Future Work
While powerful, TransFill relies on SIFT features for initial alignment; if images are extremely dark or lack texture, the alignment may fail. It also struggles with extreme perspective distortions or semantic lighting changes (e.g., day-to-night transitions). Future work may involve replacing SIFT with more robust deep-feature matchers and deeper integration of the single-image inpainting module into the end-to-end training process.
Summary updated