Anywhere: A Multi-Agent Framework for User-Guided, Reliable, and Diverse Foreground-Conditioned Image Generation

Tianyidan Xie, Rui Ma, Qian Wang, Xiaoqian Ye, Feixuan Liu, Ying Tai, Zhenyu Zhang, Lanjun Wang, Zili Yi

TL;DR

Anywhere is a novel multi-agent framework that moves away from traditional end-to-end models to solve the "foreground-conditioned image generation" problem—creating a realistic background for a specific object. By orchestrating specialized agents for semantic analysis, creative prompting, and integrity protection, it significantly improves image fidelity, diversity, and adherence to user prompts compared to state-of-the-art inpainting models.

What Problem Does This Paper Address?

Imagine you have a photo of a chair and want to see it in a "modern living room at sunset." Current image generation models often fail at this task in frustrating ways. They might accidentally "grow" extra legs on the chair (violating object integrity), place the chair in a forest when you asked for a room (foreground-background inconsistency), or generate the same boring white room every time (limited diversity).

The authors argue these failures stem from the standard end-to-end approach. When a single network tries to handle visual semantics, spatial relationships, and text prompts all at once, these signals often interfere with each other. Furthermore, training on automatically labeled datasets often introduces "noisy" masks that teach the model to blur the lines between the foreground object and the background.

Comparison of Anywhere with existing models showing common failure modes like violated integrity and inconsistency.

Methods

Instead of one giant model, Anywhere uses a modular pipeline of specialized agents:

  1. Prompt Generation Module:
    • A Foreground Analyzer (Visual Language Model or VLM) extracts detailed attributes of the object (color, material, pose).
  2. A Prompt Creator (Large Language Model or LLM) uses these details to brainstorm creative scene descriptions.
  3. A Prompt Selector (LLM) picks the most compatible prompt for the generation phase.
  4. Image Generation Module:
    • A Template Generator uses ControlNet (edge-guided diffusion) to create an initial background layout.
  5. The Template Repainter is a critical safety agent. It compares the generated background against the original object mask to detect "integrity violations" (e.g., the background bleeding into the object). If it finds a mismatch, it selectively inpaints the area to fix it.
  6. An Image Enhancer (SDXL Refiner) polishes the final composite for high-resolution harmony.
  7. Quality Evaluator: A VLM-based agent acts as a "critic," assessing the final image for lighting consistency and realism. If the quality is low, it provides feedback to the Prompt Creator to trigger a re-generation loop.
Figure 2: Overview of the Anywhere framework. (a) Our approach comprises three main components: the Prompt Generation Module, the Image Generation Module, and the Quality Evaluator (Agent). The Prompt Generation Module uses a Foreground Analyzer (VLM) to extract textual descriptions from the foreground and a Prompt Creator (LLM) to generate multiple textual prompts based on the foreground descriptions and the user textual inputs if provided. The multiple textual prompts are then assessed by the Prompt Selector (LLM) and the best matched prompt will be selected. The Image Generation module includes a Template Generator (edge-guided image generation model) that generates a template image based on the textual prompt, a Template Repainter that detects object integrity violations (highlighted in green) and resolves the violations if needed, and an Image Enhancer (high-resolution image refinement Model) to paste-back the foreground and harmonize the final output. The Quality Evaluator Agent (VLM) assesses the resulting image, providing descriptive feedback and triggering re-generation when needed. (b) Illustration of the Template Repainter that performs violation detection by foreground segmentation and mask contrasting, and inpaints violated regions if they exist. (c) Illustration of template repainting tools used in the framework.

Datasets

The researchers curated a test dataset of 3,000 images to ensure diversity:

  • 1,500 images from the LAION dataset (representing "in-the-wild" scenarios).
  • 1,500 images from the MSCOCO dataset (standard computer vision benchmark). For each image, they extracted a randomly chosen foreground segment to serve as the conditioning input.

Key Insights and Ideas

  • Decomposition is Key: By separating "understanding the object" from "generating the scene," the framework avoids the mutual interference between visual and textual prompts that plagues end-to-end models.
  • The "Repainter" as a Safeguard: Traditional inpainting often struggles with mask accuracy. The Template Repainter’s iterative detection-and-correction loop (shown in Figure 4) ensures the foreground object remains untouched and physically plausible.
  • LLMs as Creative Directors: Using an LLM to expand a simple user prompt into a rich scene description significantly boosts the diversity of the output, moving beyond the stereotypical backgrounds found in training data.
Figure 4: Ablation studies on the Template Repainter.

Key Results

Anywhere was compared against top-tier models like BrushNet, HD-Painter, and Stable Diffusion Inpainting. Quantitative and qualitative results showed:

  • Fidelity: A 4.6% improvement in FID (Fréchet Inception Distance), indicating more natural-looking images.
  • User Preference: An average 24% increase in user preference scores.
  • Reliability: A 44% reduction in "bad cases" (images with artifacts or logical errors).
  • Diversity: A 33% boost in the diversity score of generated backgrounds.
  • Text Alignment: A 5% increase in text-image matching scores for user-guided scenarios.
Figure 3: We compare our approach to advanced inpainting models on foreground-conditioned image generation tasks in both text-free (I2I) and text-guided (TI2I) scenarios. These results are generated using unconstrained, in-the-wild foreground images. Red color indicates missing elements in generated images. The inpainting models used for comparison include HD-Painter (HDP), BrushNet (BN), and Stable Diffusion 2.0 Inpainting (SDI).

Limitations and Future Work

The primary trade-off for this reliability is computational cost. Because it involves multiple model passes and a potential feedback loop, Anywhere requires roughly 2–3x more GPU time per generation than end-to-end models.

Additionally, the authors noted that the framework still struggles with "corner cases," such as objects with transparent or semi-transparent parts (like glassware), where the boundary between foreground and background is physically complex. Future work will focus on optimizing efficiency and handling these difficult "long-tail" visual scenarios.

Summary updated

Abstract

Recent advancements in image-conditioned image generation have demonstrated substantial progress. However, foreground-conditioned image generation remains underexplored, encountering challenges such as compromised object integrity, foreground-background inconsistencies, limited diversity, and reduced control flexibility. These challenges arise from current end-to-end inpainting models, which suffer from inaccurate training masks, limited foreground semantic understanding, data distribution biases, and inherent interference between visual and textual prompts. To overcome these limitations, we present Anywhere, a multi-agent framework that departs from the traditional end-to-end approach. In this framework, each agent is specialized in a distinct aspect, such as foreground understanding, diversity enhancement, object integrity protection, and textual prompt consistency. Our framework is further enhanced with the ability to incorporate optional user textual inputs, perform automated quality assessments, and initiate re-generation as needed. Comprehensive experiments demonstrate that this modular design effectively overcomes the limitations of existing end-to-end models, resulting in higher fidelity, quality, diversity and controllability in foreground-conditioned image generation. Additionally, the Anywhere framework is extensible, allowing it to benefit from future advancements in each individual agent.
BibTeX
@inproceedings{xie2025anywhere,
  author = {Tianyidan Xie and Rui Ma and Qian Wang and Xiaoqian Ye and Feixuan Liu and Ying Tai and Zhenyu Zhang and Lanjun Wang and Zili Yi},
  title = {{Anywhere: A Multi-Agent Framework for User-Guided, Reliable, and Diverse Foreground-Conditioned Image Generation}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.1609/aaai.v39i7.32797},
}
Figure