TL;DR
TransFill is a novel framework for reference-guided image inpainting that fills "holes" in a target image by intelligently warping and color-matching a second reference image. By using a multi-homography approach to handle 3D parallax and a deep color-spatial transformer, it achieves state-of-the-art results even when the two images have significant viewpoint shifts or lighting differences.
What Problem Does This Paper Address?
Image inpainting—the art of filling in missing pixels—is a staple of photo editing, used for everything from removing a distracting tourist to fixing a corrupted file. Traditional methods usually look for similar patches within the same image or use generative AI to "hallucinate" what might be there. However, these methods often fail when the hole is large or the scene is complex.
Reference-guided inpainting offers a solution: if you have another photo of the same scene taken from a slightly different angle or time, why not just copy the missing data from there? While this sounds simple, it is technically grueling. Real-world photos suffer from parallax (objects shifting differently based on their depth), different exposures, and varying white balances. Existing tools often struggle to align these images perfectly, leading to "ghosting" artifacts or jarring color seams.
Methods
The authors propose a pipeline called TransFill, which breaks the problem into four distinct stages:
- Multi-homography Proposals: Instead of trying to find one single "warp" to align the images, TransFill estimates multiple homographies. It uses a monocular depth estimator to cluster feature points into different depth planes. Each plane gets its own alignment proposal, ensuring that both the foreground and background can be aligned accurately.
- Color-Spatial Transformer (CST): Even with good homographies, small misalignments and color differences remain. The CST module uses deep bilateral filtering to adjust colors and a learned per-pixel warping field to refine the spatial alignment. Importantly, this happens at a lower resolution to maintain "rigidity" and texture, then up-samples the results.
- Single-Proposal Fusion (SPF): Each refined proposal is independently compared against a single-image inpainting reference (using a pre-trained model called ProFill). This step generates a confidence map for each proposal, identifying which parts of the reference image are actually useful.
- Multi-Proposal Fusion (MPF): Finally, a UNet-based module merges all the proposals and the single-image inpainting result into one seamless final image.
Datasets
The model was trained and evaluated using:
- RealEstate10K: A large-scale dataset of video frames from YouTube real estate tours.
- MIT-Adobe5K: Used to synthesize color inconsistencies for training color harmonization.
- User-provided Pairs: A collection of 3,000 real-world image pairs to test the model's generalization to uncalibrated, "in-the-wild" data.
Key Insights and Ideas
- Depth Matters for Alignment: A single 2D warp cannot account for a 3D world. By clustering alignment points by depth, the model creates a "multi-plane" representation that handles complex 3D scenes much better than global alignment methods.
- Decoupling Color and Space: By learning color and spatial transformations simultaneously but through distinct sub-networks, the model avoids the "blurry" results often seen when a single network tries to do too much at once.
- The Power of Sparsity: The authors found that using the SPF module to create a "bottleneck" before the final fusion was crucial. It forces the model to learn sparse, high-confidence weights, preventing the final output from becoming a muddy average of all proposals.
Key Results
TransFill significantly outperformed existing baselines like APAP (stitching) and OPN (video inpainting).
- Quantitative: On the RealEstate10K test set, TransFill achieved a PSNR of 38.83 and an SSIM of 0.9914, notably higher than the next best baseline (DFG at 36.17 PSNR).
- Qualitative: In a user study, users preferred TransFill's results over other methods between 81% and 95% of the time. It proved particularly robust at handling 90-degree rotations and extreme lighting mismatches.
Limitations and Future Work
While powerful, TransFill isn't magic. The authors note several limitations:
- Feature Dependency: If the scene is extremely dark or textureless, SIFT feature matching fails, and the alignment collapses.
- Extreme Viewpoints: The homography-based approach assumes the camera hasn't moved too far. Extreme changes in perspective still result in artifacts.
- Lighting Extremes: While it handles exposure well, it struggles with fundamental lighting changes, such as trying to fill a "daytime" hole with "nighttime" reference content.
Summary updated