CustAny: Customizing Anything from A Single Example

Abstract

CustAny is a novel zero-shot framework designed to customize any object within text-to-image models using only a single reference image. By leveraging a massive new dataset, MC-IDC, and a specialized decoupling architecture, CustAny achieves high identity fidelity across diverse categories while maintaining strong responsiveness to text-based prompts, all without the need for time-consuming fine-tuning.

Introduction

Personalizing text-to-image models like Stable Diffusion typically requires either object-specific fine-tuning (e.g., DreamBooth) or specialized zero-shot models limited to specific domains like human faces. CustAny addresses the challenge of General Object Customization, enabling users to insert unique objects into new scenes while avoiding "ID leakage"—a common issue where irrelevant details from the reference photo, such as background or lighting, interfere with the generated output.

Methodology

The CustAny framework is built upon three core technical innovations designed to bridge the gap between identity preservation and text responsiveness:

1. General ID Extractor: To capture both structural and color details, CustAny combines DINOv2 (excellent for structure) and MAE (Masked Autoencoders, better for color palettes). This hybrid approach overcomes the color-insensitivity often found in DINOv2-based models, leading to much better color fidelity.
2. Dual-Level ID Injection: Identity features are injected into the Diffusion UNet at both global (semantic) and local (fine-grained patch) levels. Global injection merges semantic features into text embeddings, while local injection uses cross-attention layers to preserve intricate details.
3. ID-Aware Decoupling Module: This module acts as the "secret sauce" by training the model to separate "Identity" from "Non-Identity" elements like pose, orientation, or motion. This prevents the model from simply copying the reference image's composition, allowing the text prompt to truly drive the action.

Expand

Source, p. 1

The MC-IDC Dataset

A major contribution of this work is the Multi-Category ID-Consistent (MC-IDC) dataset. It contains 315,000 text-image samples across 10,000 categories, including humans, animals, clothes, furniture, and tools. By using video frames and multi-view data to create pairs of images showing the same object in different states, the dataset teaches the model to recognize a universal concept of identity across different angles and motions.

Experimental Results

CustAny significantly outperforms existing state-of-the-art models like IP-Adapter, PhotoMaker, and InstantID. In general object customization, it improved the FID (Fréchet Inception Distance) score from 70.32 to 47.09 compared to IP-Adapter, while achieving higher CLIP-i and DINO-i scores for identity fidelity.

Notably, CustAny excels in specialized domains despite being a generalist model:

• Virtual Try-on: It outperforms dedicated models like MagicClothing.
• Human Customization: It achieves higher face similarity than specialized models like PhotoMaker.

Expand

Source, p. 1

Conclusion and Future Work

CustAny establishes a new benchmark for zero-shot object customization by successfully decoupling identity from pose and environment. While it currently focuses on single-object interaction, future work may expand its capabilities to multi-object scenes and the replication of extremely high-frequency textures. The MC-IDC dataset also provides a foundation for future identity-consistent video generation models.

@inproceedings{kong2025custany,
  author = {Lingjie Kong and Kai Wu and Chengming Xu and Xiaobin Hu and Wenhui Han and Jinlong Peng and Donghao Luo and Mengtian Li and Jiangning Zhang and Chengjie Wang and Yanwei Fu},
  title = {{CustAny: Customizing Anything from A Single Example}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2025},
  doi = {10.1109/cvpr52734.2025.01948},
}