One-Shot Generative Personalization and Internal Learning

This direction is no longer present in the current build. Its last version stays readable and citable here.

Version 1 of 139 papers41 conceptsfrozen 2026-10-04

Standard generative models are data-hungry, typically requiring thousands of images to learn a single concept. However, a growing body of research focuses on the "single-image" regime: the ability to capture the essence of a unique object, person, or scene from just one reference. This field splits into two primary directions: internal learning, where a model learns the statistical "DNA" of a single image from scratch, and one-shot personalization, where a pre-trained model is surgically adapted to recognize a new individual concept.

The Problem of Overfitting and Language Drift

When a model like Stable Diffusion is fine-tuned on a single image, it faces a fundamental "overfitting" trap. Because the data is so sparse, the model tends to memorize the entire image—including the background, lighting, and specific pose—rather than isolating the subject itself. Amil Dravid, Kfir Aberman, and Yossi Gandelsman observe in Interpreting the Weight Space of Customized Diffusion Models that this prevents the model from generalizing the subject into new contexts.

Furthermore, Ligong Han and colleagues identify "language drift" in SINE: SINgle Image Editing with Text-to-Image Diffusion Models. As the model learns the new concept, it often "forgets" how to follow general text prompts, making it impossible to generate the subject in diverse scenarios (e.g., "a photo of [Subject] on the moon").

Internal Learning through Patch Statistics

Before the rise of large-scale text-to-image models, researchers focused on "internal learning"—training a model to understand the recurring patterns within a single image. Vladimir Kulikov and Shahar Yadin demonstrate this in SinDDM: A Single Image Denoising Diffusion Model by using a multi-scale, patch-based architecture.

The intuition is that a single image contains a wealth of information if you look at its patches. By training on different crops and resolutions, the model learns the local texture and global structure of that specific scene. Tobias Hinz and others refined this with ConSinGAN, which trains multiple resolution stages concurrently to maintain global coherence while capturing fine details.

Personalization via Concept Disentanglement

To make one-shot learning work in pre-trained models, researchers must "disentangle" the subject from its surroundings. Chaohui Yu and Zhen Zhu 0006 propose SingleInsert, which uses a two-stage process to isolate the subject from the background. This ensures that when you ask the model for the subject in a new setting, it doesn't bring the original background along with it.

Other researchers use clever analogies to guide the model. In FaceChain-SuDe, Pengchong Qiao and Lei Shang treat the specific subject as a "derived class" in an Object-Oriented Programming sense. The subject inherits general traits from a "base class" (like "Dog") but maintains its unique identity. This allows the model to generate the subject performing actions it never saw in the single reference image, like a specific dog running or barking.

Anchoring Identities in Known Spaces

A powerful shortcut for one-shot learning is to map the new, unknown identity onto a space the model already understands. Ge Yuan introduces Celeb Basis, which represents a new person as a linear combination of celebrity faces the model already knows. By only learning a few weights to mix these "celebrity anchors," the model can capture a new identity with extreme efficiency.

Similarly, Yu-Chuan Su and Kelvin C.K. Chan developed an Identity Encoder that bypasses fine-tuning entirely. Instead of optimizing the model for minutes, a feed-forward network "looks" at the reference image and instantly produces a representation that the diffusion model can use to generate the person in any context.

Balancing Fidelity and Editability

The ultimate challenge in this research is the trade-off between fidelity (how much the output looks like the reference) and editability (how well the model follows new prompts). Pengxu Wei and Liang Lin address this in DreamArtist by training a "Positive-Negative Adapter." This framework learns what the subject is and what it is not simultaneously, providing a clearer boundary for the model to follow during generation.

By combining these strategies—patch-based internal learning, semantic disentanglement, and identity anchoring—researchers are moving toward a future where a single photograph is enough to unlock a world of personalized, high-fidelity generative content.

Papers in this version

History & citationVersion 1 · 2026-10-04

No longer present in the current build · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_one_shot_generative_personalization_and__ca370920,
  title = {One-Shot Generative Personalization and Internal Learning},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/one-shot-generative-personalization-and-internal-learning/rdr_ca370920266240a989c1b121c20f6cd4},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-21},
}

This exact version (version 1, frozen text):

@misc{lacuna_one_shot_generative_personalization_and__ca370920_v1,
  title = {One-Shot Generative Personalization and Internal Learning},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/one-shot-generative-personalization-and-internal-learning/rdr_ca370920266240a989c1b121c20f6cd4/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-21},
}