FaceChain-SuDe: Building Derived Class to Inherit Category Attributes for One-shot Subject-Driven Generation

Pengchong Qiao, Lei Shang, Chang Liu, Baigui Sun, Xiangyang Ji, Jie Chen

TL;DR

FaceChain-SuDe treats subject-driven image generation as an object-oriented programming (OOP) problem, modeling a specific subject as a derived class that inherits general attributes from its semantic base class (category). By introducing a plug-and-play regularization term, it enables high-fidelity personalization from just a single example image while successfully generating complex actions and attributes that the baseline models typically "forget."

What Problem Does This Paper Address?

Subject-driven generation aims to personalize text-to-image models so they can generate a specific subject (e.g., your pet dog "Spike") in new contexts. Current "one-shot" methods—which use only a single example image—often suffer from a lack of imaginative variety.

If your only photo of Spike shows him sitting, the model often fails to generate Spike "running" or "jumping." The authors argue this happens because the model treats the new subject as an isolated concept rather than a specialization of an existing category. In essence, the model learns Spike's "private attributes" (his specific fur pattern) but loses access to the "public attributes" of the "Dog" category (the ability to run or bark).

The failure of vanilla baselines to generate category-specific actions like running or jumping from a single image.

Methods

The authors propose Subject-Derived regularization (SuDe), a method inspired by the concept of inheritance in OOP.

  1. Class Modeling: They model the semantic category (e.g., "Dog") as a base class containing public attributes provided by the pre-trained model. The specific subject ("Spike") is a derived class that inherits these public attributes while learning private, subject-specific attributes from the user's image.
  2. Implicit Diffusion Classifier: Instead of adding a separate, external classifier (which might have misaligned semantics), SuDe leverages the diffusion model's own internal knowledge. Using Bayes' theorem, they derive a "Subject Derivation Loss" from the model's denoising predictions. This loss constrains the generated images of the subject to remain semantically recognizable as members of the base category.
  3. Loss Truncation: To prevent the model from over-optimizing and causing the latent space to collapse (a common issue when forcing high classification probability on noisy images), they introduce a truncation strategy. This ensures the regularization only applies when the classification probability is below a natural threshold.
The SuDe pipeline: (a) learning private attributes via reconstruction and (b) inheriting public attributes via the Subject-Derived loss.

Datasets

The researchers evaluated their method using the DreamBench dataset, which consists of 30 subjects across 15 categories (e.g., dogs, teapots, backpacks).

  • One-Shot Setup: Unlike standard benchmarks that use multiple images, they restricted training to just one image per subject to test the limits of the method.
  • Attribute-Related Prompts: They curated a new set of 5 attribute-rich prompts for each subject (e.g., "a running {S}", "a clear {S}") to specifically measure how well the model inherits category behaviors.

Key Insights and Ideas

  • Personalization as Specialization: The core "aha" moment is that subject-driven generation shouldn't just be about reconstruction; it's about inheritance. A personalized dog is still a dog and should behave like one.
  • The Model is its Own Best Critic: By deriving the classifier directly from the diffusion model's denoising math, the authors ensure that the regularization is perfectly aligned with the model's pre-trained priors.
  • Stability through Truncation: The discovery that over-optimizing category probability leads to "generation collapse" (Figure 5) highlights the delicate balance required when fine-tuning diffusion models.
Comparison showing how SuDe improves attribute alignment (e.g., "running," "playing ball") across different baselines.

Key Results

SuDe was tested as a plug-and-play module for three major baselines: DreamBooth, Custom Diffusion, and ViCo.

  • Quantitative Gains: SuDe consistently improved BLIP-T scores (a metric for attribute-text alignment). For example, on the ViCo baseline using Stable Diffusion v1.4, it provided a +4.2% boost in attribute alignment.
  • Qualitative Improvements: As seen in Figure 3, SuDe allows a "Cartoon Character" to actually run or a "Teapot" to appear made of glass, whereas vanilla baselines often ignore these prompts and simply reproduce the original sitting pose or material.
  • Fidelity Preservation: Importantly, these gains in imagination did not come at the cost of subject fidelity (measured by DINO-I), meaning Spike still looks like Spike.

Limitations and Future Work

  • Inherent Backbone Failures: SuDe inherits the flaws of its base model. If Stable Diffusion struggles to render text correctly, SuDe cannot fix that (e.g., blurry text on a "Can" subject).
  • Indirect Attributes: The method struggles with attributes that aren't inherent to the category. For instance, "a dog wearing a yellow shirt" is difficult because "wearing a shirt" isn't a standard attribute of the "Dog" base class.
  • Hyperparameter Sensitivity: The weight of the SuDe loss ($w_s$) needs to be tuned; too high a weight can dilute the subject's private attributes, leading to a loss of identity.
Visualizing the impact of the training weight $w_s$ on the balance between category inheritance and subject fidelity.

Summary updated

Abstract

Subject-driven generation has garnered significant interest recently due to its ability to personalize text-to-image generation. Typical works focus on learning the new subject's private attributes. However, an important fact has not been taken seriously that a subject is not an isolated new concept but should be a specialization of a certain category in the pre-trained model. This results in the subject failing to comprehensively inherit the attributes in its category, causing poor attribute-related generations. In this paper, motivated by object-oriented programming, we model the subject as a derived class whose base class is its semantic category. This modeling enables the subject to inherit public attributes from its category while learning its private attributes from the user-provided example. Specifically, we propose a plug-and-play method, Subject-Derived regularization (SuDe). It constructs the base-derived class modeling by constraining the subject-driven generated images to semantically belong to the subject's category. Extensive experiments under three baselines and two backbones on various subjects show that our SuDe enables imaginative attribute-related generations while maintaining subject fidelity. Codes will be open sourced soon at FaceChain (https://github.com/modelscope/facechain).
BibTeX
@inproceedings{qiao2024facechain,
  author = {Pengchong Qiao and Lei Shang and Chang Liu and Baigui Sun and Xiangyang Ji and Jie Chen},
  title = {{FaceChain-SuDe: Building Derived Class to Inherit Category Attributes for One-shot Subject-Driven Generation}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2024},
}
Figure