Anywhere: A Multi-Agent Framework for Reliable and Diverse Foreground-Conditioned Image Inpainting

Tianyidan Xie, Rui Ma, Qian Wang, Xiaoqian Ye, Feixuan Liu, Ying Tai, Zhenyu Zhang, Zili Yi

Introduction

Image generation technology has advanced rapidly, yet the specific task of foreground-conditioned image generation—creating a realistic background for a single, isolated object—remains a significant challenge. Traditional end-to-end inpainting models often struggle to maintain the integrity of the foreground object or produce contextually relevant and diverse environments. The "Anywhere" framework addresses these issues by transitioning from a monolithic "black box" approach to a modular, multi-agent system. By coordinating specialized agents for semantic analysis, creative prompt generation, and iterative quality control, Anywhere achieves superior results in object fidelity and creative diversity.

The Limitations of Current Inpainting Models

Existing state-of-the-art models, such as BrushNet and HD-Painter, frequently encounter four primary failure modes when tasked with background generation:

  1. Violated Object Integrity: Inaccurate segmentation masks often lead models to "bleed" into or unnaturally extend the foreground object.
  2. Foreground-Background Inconsistency: Models may generate semantically mismatched environments, such as placing a snow-based object in a tropical setting.
  3. Limited Diversity: There is a tendency to produce safe, stereotypical backgrounds rather than creative variations.
  4. Compromised Textual Consistency: Balancing the visual constraints of the object with specific user-provided text prompts often results in one being ignored.
Anywhere addresses common failure modes of existing inpainting models, such as integrity violations and semantic inconsistencies.

The Anywhere Framework: A Multi-Agent Approach

The core innovation of Anywhere is its modular architecture, which decomposes the generation process into three distinct stages managed by specialized agents.

The Anywhere framework architecture consists of three main modules: Prompt Generation, Image Generation, and a Quality Evaluator loop.

1. Prompt Generation Module

This stage focuses on semantic understanding and creative brainstorming. A Foreground Analyzer (using a Vision Language Model) extracts detailed attributes like pose and material. Subsequently, a Prompt Creator (LLM) generates multiple diverse background candidates, which are then filtered by a Prompt Selector to ensure compatibility and realism.

2. Image Generation Module

The generation process begins with a Template Generator using edge-guided diffusion to respect object boundaries. A critical component here is the Template Repainter, which acts as a specialized auditor. It identifies areas where the generation process may have "leaked" into the foreground by comparing the intended mask with the generated result, iteratively fixing these artifacts. Finally, an Image Enhancer (SDXL-based) refines the composite for high-resolution harmony.

3. Quality Evaluator

To ensure the final output meets high standards, a VLM-based Quality Evaluator performs a "vibe check," assessing lighting, spatial logic, and semantic relevance. If the image is deemed suboptimal, the agent provides descriptive feedback to the Prompt Generation Module for a re-generation attempt, effectively closing the loop on the creative process.

Experimental Results

The framework was evaluated using a curated dataset of 3,000 images from LAION and MSCOCO. Evaluation covered both text-free and text-guided scenarios, comparing Anywhere against leading models like BrushNet and Stable Diffusion Inpainting.

Key performance metrics include:

  • Fidelity: A 4.6% improvement in Fréchet Inception Distance (FID).
  • Human Preference: A 24% higher preference score from human evaluators.
  • Reliability: A 44% reduction in "bad cases" involving glitches or illogical content.
  • Diversity: A 33% increase in diversity scores, reflecting the framework's ability to explore creative backgrounds.
  • Text Alignment: A 5% improvement in text-image matching for prompt-guided tasks.

Conclusion and Future Directions

The Anywhere framework demonstrates that complex semantic tasks in image generation are often better handled by a committee of specialized agents than by a single monolithic model. While the modular approach significantly improves quality and reliability, it comes with a computational trade-off, requiring approximately 2–3 times more GPU time than end-to-end models. Future research will focus on optimizing this efficiency and improving the handling of complex scenarios, such as transparent objects where the background must be visible through the foreground.

Summary updated

Abstract

Recent advancements in image inpainting, particularly through diffusion modeling, have yielded promising outcomes. However, when tested in scenarios involving the completion of images based on the foreground objects, current methods that aim to inpaint an image in an end-to-end manner encounter challenges such as "over-imagination", inconsistency between foreground and background, and limited diversity. In response, we introduce Anywhere, a pioneering multi-agent framework designed to address these issues. Anywhere utilizes a sophisticated pipeline framework comprising various agents such as Visual Language Model (VLM), Large Language Model (LLM), and image generation models. This framework consists of three principal components: the prompt generation module, the image generation module, and the outcome analyzer. The prompt generation module conducts a semantic analysis of the input foreground image, leveraging VLM to predict relevant language descriptions and LLM to recommend optimal language prompts. In the image generation module, we employ a text-guided canny-to-image generation model to create a template image based on the edge map of the foreground image and language prompts, and an image refiner to produce the outcome by blending the input foreground and the template image. The outcome analyzer employs VLM to evaluate image content rationality, aesthetic score, and foreground-background relevance, triggering prompt and image regeneration as needed. Extensive experiments demonstrate that our Anywhere framework excels in foreground-conditioned image inpainting, mitigating "over-imagination", resolving foreground-background discrepancies, and enhancing diversity. It successfully elevates foreground-conditioned image inpainting to produce more reliable and diverse results.
BibTeX
@inproceedings{xie2025anywhere,
  author = {Tianyidan Xie and Rui Ma and Qian Wang and Xiaoqian Ye and Feixuan Liu and Ying Tai and Zhenyu Zhang and Zili Yi},
  title = {{Anywhere: A Multi-Agent Framework for Reliable and Diverse Foreground-Conditioned Image Inpainting}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.1609/aaai.v39i7.32797},
}
Figure