FreeInpaint: Tuning-free Prompt Alignment and Visual Rationality Enhancement in Image Inpainting

Chao Gong, Dong Li, Yingwei Pan, Jingjing Chen, Ting Yao, Tao Mei

TL;DR

FreeInpaint is a tuning-free, plug-and-play framework that significantly improves text-guided image inpainting by directly optimizing diffusion latents on the fly during inference. By combining initial noise optimization to steer model attention with a decomposed guidance strategy using off-the-shelf reward models, it simultaneously solves the dual challenges of prompt alignment and visual coherence without requiring any model retraining.

What Problem Does This Paper Address?

Text-guided image inpainting allows users to clear out an area of an image and fill it with something new based on a text prompt. While modern text-to-image (T2I) diffusion models are incredibly powerful, adapting them for inpainting introduces a frustrating trade-off between prompt alignment (actually generating what the user asked for) and visual rationality (making the new object blend seamlessly and logically with the unmasked background).

Existing training-based methods often ignore the text prompt because they are trained on random masks where the background context alone is enough to reconstruct the image. On the flip side, training-free methods that force prompt alignment often introduce ugly artifacts or unnatural boundaries. FreeInpaint addresses this dilemma without requiring any expensive fine-tuning or specialized training datasets.

Methods

FreeInpaint introduces two core components that operate entirely during the inference phase:

  1. Prior-Guided Noise Optimization (PriNo): Diffusion models are notoriously sensitive to their initial random noise ($z_T$). PriNo optimizes this initial noise at the very first denoising step ($t = t_{ini}$). By utilizing a custom attention loss, it forces both cross-attention (text-to-visual) and self-attention (inter-patch visual) maps to concentrate heavily inside the masked region. This ensures that the structure of the prompted object is locked in early on, preventing the model from ignoring the prompt or letting the background overwrite it.
  2. Decomposed Training-free Guidance (DeGu): During the subsequent denoising steps, FreeInpaint mathematically decomposes the conditional distribution of the inpainting process into three distinct objectives: text alignment, visual rationality, and human preference. It then leverages specialized, off-the-shelf reward models (such as local CLIPScore for text, InpaintReward for visual coherence, and ImageReward for aesthetic preference) to compute gradients and refine the predicted noise at each step.

The complete framework is illustrated below:

FreeInpaint Framework Overview

Datasets

The authors evaluated FreeInpaint using two primary benchmarks:

  • EditBench: A standard benchmark featuring handcrafted, scribble-like free-form masks designed to test realistic image editing capabilities.
  • Re-annotated MSCOCO: Standard MSCOCO validation labels consist of single words, which are inadequate for testing complex text-guided inpainting. The authors used LLaVA-NeXT to generate detailed, mask-level captions for cropped objects, curating a high-quality evaluation dataset of 300 image-mask-description triplets.

Key Insights and Ideas

  • The First Step Dictates the Future: The authors observed that attention maps at the very first denoising step ($t = t_{ini}$) closely mirror the average attention maps across the entire diffusion process. This insight allows PriNo to optimize initial noise using only the first step's gradients, making it computationally efficient.
  • Divide and Conquer the Trade-off: Trying to optimize for prompt alignment and background consistency simultaneously in a single formula leads to a tug-of-war. Decoupling them—using PriNo for structural alignment at the start and DeGu for fine-grained detail and coherence during denoising—elegantly resolves the trade-off.
  • Flipping the Guidance Modulator: Instead of using the standard noise-based guidance weight ($\sqrt{1 - \bar{\alpha}_t}$), FreeInpaint uses $\sqrt{\bar{\alpha}_t}$ as a reward modulator. Since $\sqrt{\bar{\alpha}_t}$ increases monotonically as denoising progresses, it naturally down-weights the influence of unreliable reward predictions during the early, highly chaotic noisy steps.

Key Results

FreeInpaint was thoroughly tested across various base architectures, including CNN-based U-Nets (SD1.5, PowerPaint, BrushNet, SDXL) and Diffusion Transformers (SD3).

  • Quantitative Performance: On EditBench, FreeInpaint consistently boosted performance across all baselines. For instance, when integrated with BrushNet, it improved the ImageReward score from 0.2729 to 0.5006 and local CLIPScore from 26.45 to 27.81, while simultaneously lowering the LPIPS distance (indicating better visual quality).
  • User Study: In a blind user study with 59 evaluators, FreeInpaint achieved a dominant 64.52% win rate for overall quality and prompt alignment, vastly outperforming standard Stable Diffusion Inpainting (16.16%) and its closest training-free competitor, HD-Painter (19.32%).
Qualitative Comparisons

Limitations and Future Work

  • Extremely Small Masks: FreeInpaint still struggles when the masked area is exceptionally small. In these scenarios, the background context dominates, and the model tends to fill the mask with background textures rather than generating the requested object. The authors propose exploring hierarchical attention steering to mitigate this in future work.
  • Proxy Bias: Because DeGu relies on pre-trained reward models to simulate human preference, any inherent biases or blind spots within those reward models can potentially influence the final generated output.

Summary updated

Abstract

Text-guided image inpainting endeavors to generate new content within specified regions of images using textual prompts from users. The primary challenge is to accurately align the inpainted areas with the user-provided prompts while maintaining a high degree of visual fidelity. While existing inpainting methods have produced visually convincing results by leveraging the pre-trained text-to-image diffusion models, they still struggle to uphold both prompt alignment and visual rationality simultaneously. In this work, we introduce FreeInpaint, a plug-and-play tuning-free approach that directly optimizes the diffusion latents on the fly during inference to improve the faithfulness of the generated images. Technically, we introduce a prior-guided noise optimization method that steers model attention towards valid inpainting regions by optimizing the initial noise. Furthermore, we meticulously design a composite guidance objective tailored specifically for the inpainting task. This objective efficiently directs the denoising process, enhancing prompt alignment and visual rationality by optimizing intermediate latents at each step. Through extensive experiments involving various inpainting diffusion models and evaluation metrics, we demonstrate the effectiveness and robustness of our proposed FreeInpaint.
BibTeX
@inproceedings{gong2026freeinpaint,
  author = {Chao Gong and Dong Li and Yingwei Pan and Jingjing Chen and Ting Yao and Tao Mei},
  title = {{FreeInpaint: Tuning-free Prompt Alignment and Visual Rationality Enhancement in Image Inpainting}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2026},
  url = {https://openreview.net/forum?id=y5G0Ldryy4},
}
Figure