pOps: Photo-Inspired Diffusion Operators

Elad Richardson, Yuval Alaluf, Ali Mahdavi-Amiri, Daniel Cohen-Or

TL;DR

pOps (Photo-Inspired Diffusion Operators) is a framework that treats image generation like a mathematical equation. By training specific semantic "operators" directly within the CLIP image embedding space, it allows users to perform precise manipulations—like applying a specific texture to an object or merging two distinct scenes—with far more control and predictability than standard text prompting.

What Problem Does This Paper Address?

While text-to-image models like Stable Diffusion and DALL-E have revolutionized content creation, they remain notoriously difficult to "steer." Achieving a specific result often requires grueling hours of prompt engineering.

The core issue is that language is often too imprecise to describe complex visual relationships. Researchers have previously noted that the CLIP image embedding space (the mathematical representation of what an image "means") is semantically meaningful—for example, averaging the embeddings of two images sometimes yields a logical hybrid. However, this "latent averaging" is unpredictable and uncontrollable. The authors of pOps wanted to turn these haphazard interactions into reliable, learned operations: a "Union" operator that actually knows how to combine objects, or a "Texturing" operator that understands how to wrap a material around a shape.

Methods

The pOps framework is built on top of the Diffusion Prior model (originally introduced with DALL-E 2). Traditionally, a Diffusion Prior maps text embeddings to image embeddings. The authors realized they could repurpose this architecture to act as a "semantic processor."

The "Operator" Concept

Instead of feeding the model text, they feed it image embeddings as inputs. They trained several distinct operators:

  • Union: Combines objects from two different images into one scene.
  • Texturing: Takes an object from Image A and a texture patch from Image B to create a textured version of the object.
  • Scene: Places a specific object into a new background.
  • Instruct: Applies a text-based adjective (like "spiky" or "melting") directly to an image embedding.
  • Composition: Merges multiple parts (e.g., individual clothing items) into a single coherent image (a full outfit).

Training and Rendering

The team fine-tuned the transformer layers of the Diffusion Prior to perform these specific tasks. Crucially, the actual image generator (the "renderer," like Kandinsky 2.2) remains frozen. This means pOps focuses entirely on the logic of the transformation in the embedding space before any pixels are even drawn.

One of the most innovative aspects is the Generative Tree. Because these operators work in the same embedding space, they can be composed. You can take an object, apply a "melting" operator, then use a "Scene" operator to place that melting object on a beach—mirroring the "Constructive Solid Geometry" (CSG) approach used in traditional 3D modeling.

Datasets

Since "paired" data for these specific tasks (e.g., an image of a plain dog and the exact same dog but "spiky") doesn't exist in the wild, the authors created synthetic training sets:

  • Synthetic Paired Data: They used SDXL-Turbo and SDXL-Depth to generate tens of thousands of examples for the Texturing, Scene, and Union operators.
  • ATR Dataset: Used for the "Composition" operator to learn how to assemble human outfits.
  • Open Images: Used as a source for diverse object classes for the Union and Instruct operators.
  • Scale: Most operators were trained on approximately 50,000 generated samples.

Key Insights and Ideas

  • CLIP as a Workspace: Rather than just using CLIP to "check" if an image matches a prompt, pOps treats the embedding space as a playground where visual concepts can be mathematically manipulated.
  • Prior Repurposing: The paper proves that the Diffusion Prior is a versatile "reasoning" engine. It doesn't just translate text to images; it can learn the "rules" of visual composition if given the right training data.
  • Zero-Shot Compositionality: Even though the operators are trained independently, they can be chained together in "equations" to create complex scenes that would be nearly impossible to describe accurately with a single text prompt.

Key Results

  • Qualitative Superiority: Compared to simple latent averaging (which often produces blurry or nonsensical hybrids), pOps produces crisp, semantically accurate results that preserve the identity of the input objects.
  • User Preference: In human evaluations, the pOps Instruct operator was preferred 60.31% of the time over baselines like InstructPix2Pix and IP-Adapter.
  • Texturing Success: For the texturing task, pOps achieved a user preference score of 57.14%, significantly outperforming specialized methods like ZeST and VSP.
  • Cross-Model Compatibility: The authors demonstrated that a pOps-modified embedding can be "rendered" by different models (Kandinsky, IP-Adapter, or even combined with ControlNet) without further tuning.

Limitations and Future Work

  • The CLIP Bottleneck: The authors acknowledge that CLIP embeddings sometimes struggle to capture very fine-grained details, leading to "attribute leakage" (e.g., the color of one object accidentally bleeding into another in a Union operation).
  • Independent Training: Currently, each operator requires its own training run. The authors suggest that future work could involve a single "universal" operator model or using more efficient adaptation techniques like LoRA.
  • Identity Preservation: While pOps is great at semantic identity, it isn't a "perfect" replicator. It captures the idea of the input object better than its exact pixel-perfect geometry.

Summary updated

Abstract

Text-guided image generation enables the creation of visual content from textual descriptions. However, certain visual concepts cannot be effectively conveyed through language alone. This has sparked a renewed interest in utilizing the CLIP image embedding space for more visually-oriented tasks through methods such as IP-Adapter. Interestingly, the CLIP image embedding space has been shown to be semantically meaningful, where linear operations within this space yield semantically meaningful results. Yet, the specific meaning of these operations can vary unpredictably across different images. To harness this potential, we introduce pOps, a framework that trains specific semantic operators directly on CLIP image embeddings. Each pOps operator is built upon a pretrained Diffusion Prior model. While the Diffusion Prior model was originally trained to map between text embeddings and image embeddings, we demonstrate that it can be tuned to accommodate new input conditions, resulting in a diffusion operator. Working directly over image embeddings not only improves our ability to learn semantic operations but also allows us to directly use a textual CLIP loss as an additional supervision when needed. We show that pOps can be used to learn a variety of photo-inspired operators with distinct semantic meanings, highlighting the semantic diversity and potential of our proposed approach.
BibTeX
@inproceedings{richardson2025pops,
  author = {Elad Richardson and Yuval Alaluf and Ali Mahdavi-Amiri and Daniel Cohen-Or},
  title = {{pOps: Photo-Inspired Diffusion Operators}},
  booktitle = {SIGGRAPH (Conference Paper Track)},
  year = {2025},
  doi = {10.1145/3721238.3730615},
}

Loading related content...

Figure