HomeDiffusion: Zero-Shot Object Customization with Multi-View Representation Learning for Indoor Scenes

Guoqiu Li, Jin Song, Yiyun FeiAAAI 2025 · DOI

TL;DR

This paper introduces HomeDiffusion, a novel zero-shot object customization method that excels at seamlessly integrating reference objects into indoor scenes. By leveraging multi-view representations and a novel background-driven learning process, HomeDiffusion achieves superior fidelity and perspective harmony compared to existing methods, even for asymmetric objects.

What Problem Does This Paper Address?

The goal of object customization in image editing is to realistically place a new object into an existing image, ensuring it matches the scene's lighting, perspective, and spatial relationships, all while preserving the object's identity. While recent advancements in diffusion models have shown promise, they often struggle with two key challenges:

  1. Multi-view Object Representation: Existing methods often rely on single-view reference images. This makes it difficult to generate objects with accurate perspectives, especially for asymmetric objects, when the target scene's orientation differs significantly from the reference.
  2. High-Fidelity Detail Preservation: Even with powerful encoders, maintaining the fine details (textures, patterns) of the reference object during generation remains a challenge, leading to visually awkward results.

This paper aims to address these limitations by enabling zero-shot object customization with improved detail preservation and harmonious perspective alignment, particularly for indoor scenes.

Methods

HomeDiffusion employs a two-stage approach built upon the Stable Diffusion (SD) framework:

  1. Multi-view Object Representation Learning (MORL):
    • HD Visual Encoder: A novel encoder is proposed that processes object images at both global and local resolutions. It combines features from a downscaled image (global view) with features from cropped patches (local view) to capture fine details without overwhelming the model with excessive tokens.
  2. Multi-view Feature Encoding: Given multiple reference images of an object, the HD Visual Encoder extracts features. These are then processed by an MLP to create a multi-view object representation token ($C_o$).
  3. ControlNet Guidance: To guide the diffusion process towards the correct viewpoint, a Canny edge map of the target viewpoint is extracted and injected into the U-Net decoder via residual connections using a pre-trained ControlNet.
  4. Background-driven Object Customization Learning (BOCL):
    • Composited Image Guidance: A composited image ($x_c$) is created by pasting the reference object onto a masked background image at the user-specified location. This image, containing both scene context and high-fidelity object details, is used to guide the diffusion model.
  5. Pixel-Aligned Cross-Attention: To ensure that the high-fidelity details from the composited image are effectively integrated into the generated image, pixel-aligned cross-attention is applied within the U-Net decoder. This mechanism helps to preserve details even when the object's viewpoint in the composite image doesn't perfectly align with the target viewpoint.

Datasets

  • 3D-FRONT Dataset: The authors utilized the 3D-FRONT dataset, a large repository of synthetic indoor scenes, to generate their training data. They rendered scene images from various camera angles and extracted multi-view furniture images.
  • ZOC-Indoor-Eval Benchmark: A new benchmark is introduced for evaluating object customization in indoor scenes. It comprises over 180k scene images across 8 furniture categories, with each scene including a multi-view reference set and a scene file with a target location mask. The furniture in the scene image has a different viewpoint than the reference images.
  • Viton-HD Benchmark: The Viton-HD dataset was also used for quantitative evaluation, particularly for virtual try-on scenarios.

Key Insights and Ideas

  • Multi-view is Crucial for Perspective: The paper strongly emphasizes that single-view reference images are insufficient for achieving accurate perspective alignment, especially for asymmetric objects. Learning from multiple viewpoints is key to understanding and replicating an object's 3D structure.
  • Hierarchical Feature Extraction for Detail: The HD Visual Encoder's approach of combining global and local features is a smart way to balance comprehensive scene understanding with fine-grained object detail.
  • Composited Image as a "Hint": Using a composited image that already has the object placed in the scene acts as a powerful intermediate guide, helping the diffusion model understand both the object's identity and its intended placement.
  • Pixel-Aligned Cross-Attention for Detail Integration: This technique is vital for ensuring that the detailed information from the composited image is effectively transferred into the final generated image, preventing loss of fidelity.

Key Results

  • Superior Performance: HomeDiffusion outperforms existing zero-shot methods like Paint-by-Example and AnyDoor on the ZOC-Indoor-Eval benchmark, achieving higher CLIP and DINO scores. For example, it achieved a DINO score of 86.2 with single-view reference images.
  • Multi-view Advantage: The method shows increasing performance gains as more viewpoints are provided, with a DINO score of 88.1 when using five viewpoints.
  • High Fidelity: Qualitative results demonstrate that HomeDiffusion preserves object details (color, pattern) better than competitors, as seen in comparisons with DreamBooth and AnyDoor.
  • User Study Validation: A user study with interior design experts and novices showed HomeDiffusion achieved significantly higher scores for both object identity preservation (fidelity) and scene consistency (harmony).
  • Generalization: Despite being trained on indoor scenes, HomeDiffusion shows promising generalization capabilities to outdoor scenes and virtual try-on tasks (Viton-HD).

Limitations and Future Work

  • Training Data: While the 3D-FRONT dataset is extensive, the authors acknowledge that high-quality 3D indoor scene data is generally scarce, which poses challenges for research.
  • Computational Cost: While the authors mention that their method adds only a small percentage to inference time, the HD visual encoder and multi-stage process likely incur higher computational costs during training compared to simpler methods.
  • Generalization to Diverse Scenes: While generalization to outdoor scenes is shown, further exploration into more diverse and complex environments could be a future direction.
  • User Interaction: The current method relies on a binary mask for object placement. Future work could explore more intuitive user interaction methods for specifying object placement and pose.

Summary updated

BibTeX
@inproceedings{li2025homediffusion,
  author = {Guoqiu Li and Jin Song and Yiyun Fei},
  title = {{HomeDiffusion: Zero-Shot Object Customization with Multi-View Representation Learning for Indoor Scenes}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.1609/aaai.v39i5.32496},
}
Figure