TL;DR
Diffusion models frequently suffer from two operational flaws: they produce repetitive, near-identical outputs when prompted repeatedly, and they risk regurgitating exact training samples. To fix this without retraining, SPELL (SPArse rEpelLency) introduces a geometrically grounded, training-free guidance term that steers diffusion trajectories away from "shielded" reference regions. By sparsely nudging predicted endpoints only when they drift too close to protected or concurrently generated images, SPELL boosts generation diversity and prevents training data replication with minimal precision or compute overhead.
What Problem Does This Paper Address?
Despite the success of modern text-to-image and class-conditioned diffusion models, deploying them reveals two critical challenges:
- Lack of Diversity: Under standard sampling—especially when boosted by classifier-free guidance (CFG)—models often settle into narrow modes, generating only minor variations of the same composition across repeated runs with identical prompts.
- Data Memorization / Replication: Diffusion models can memorize and reproduce training samples, raising copyright, privacy, and licensing concerns.
Prior diversity solutions (such as continuous particle interactions or condition annealing) frequently disrupt the sampling trajectory throughout the entire process, harming sample fidelity or drastically slowing inference. The authors tackle how to enforce sample diversity and guarantee avoidance of specific reference sets post-hoc, efficiently, and without degrading image quality.
Methods
SPELL modifies the reverse stochastic differential equation (SDE) during generation using a simple geometric correction:
- Endpoint Prediction via Tweedie’s Formula: At any reverse step $t$, the denoiser predicts the final clean image $\hat{x}_0 = D_\theta(t, x_t) \approx \mathbb{E}[X_0 \mid X_t = x_t]$.
- Hard-Sphere Repellency: The target is protected by defining exclusion "shields" of radius $r$ around reference points $z_k$: $$\mathcal{S} = \bigcup_k \{x : \|x - z_k\|_2 \le r\}$$
-
Sparse Correction ($\Delta$): If the trajectory is predicted to land within a shield ($\|\hat{x}_0 - z_k\|_2 < r$), SPELL adds a radial repulsive term to push $\hat{x}_0$ back onto the boundary of the ball: $$\Delta = \sum_{k=1}^K \text{ReLU}\left(\frac{r}{\|\hat{x}_0 - z_k\|_2} - 1\right)(\hat{x}_0 - z_k)$$ An optional overcompensation multiplier $\lambda \ge 1$ (e.g., $\lambda = 1.6$) helps push trajectories cleanly into adjacent attraction basins earlier in the sampling process.
-
Flexible Modes:
- Intra-batch / Inter-batch Diversity: The reference points $z_k$ are dynamically populated by other images being generated in the same batch or across previous batches for the same prompt.
- Static Image Shielding: The set $\{z_k\}$ contains protected real images (e.g., copyrighted or training data). When scaling to large reference sets, SPELL couples with fast approximate nearest-neighbor search (via FAISS
IndexIVFFlat).
Datasets
- Soft-Label CC12M: A newly curated subset of Conceptual 12M comprising 5,554 distinct captions with 4 to 128 corresponding ground-truth images each (554 validation, 5,000 test captions; 41,596 total images), constructed to evaluate one-to-many diversity while filtering out generic titles, web-scraping artifacts, and identifiable individuals.
- ImageNet-1k: Used both for class-conditional generation benchmarks and as a massive static shielded set (all 1.2M training images) to test replication prevention.
- MS COCO: Used for zero-shot qualitative text-to-image evaluations on un-trained prompts.
Key Insights and Ideas
- Intervene on Future Expectations, Not Current States: Unlike Particle Guidance (which continuously repels noisy latent representations $x_t$), SPELL projects trajectories based on the expected denoised output $\hat{x}_0$.
- Sparsity by Design: Because the ReLU activation zeroes out whenever $\|\hat{x}_0 - z_k\|_2 \ge r$, SPELL is completely inactive across most steps. Trajectories headed toward distinct modes are left untouched.
- Early Action Prevents Late Artifacts: Denoising dynamics outline coarse global structures early ($t \in [0.6, 1.0]$) and refine local texture late ($t < 0.5$). SPELL triggers primarily in early timesteps (declining to only ~20% active runs by $t=0.6$), allowing the native diffusion score to refine high-frequency details later without interference or visual artifacts.
- Doob’s $h$-Transform Connection: The authors mathematically link SPELL to conditioning via Doob’s $h$-transform, showing it can be seen as an exact geometric projection alternative to softer, hard-to-tune Gaussian diffusion posterior sampling (DPS).
Key Results
- Diversity vs. Quality Pareto Fronts: Across Stable Diffusion 3, Latent Diffusion, EDMv2, MDTv2, and Simple Diffusion, SPELL consistently boosts diversity metrics (Recall, Vendi Score) with minimal impact on Precision and FID. On Latent Diffusion (CC12M), SPELL delivers better Pareto trade-offs across Recall/Precision, Coverage/Density, and Vendi/CLIP score than competing methods like CADS, Interval Guidance (IG), and Particle Guidance (PG).
- Training Data Shielding at Scale: When shielding against all 1.2 million ImageNet-1k training samples:
- Baseline EDMv2 generated images within an $L_2$ radius $r=60$ of a training image 7.60% of the time.
- Adding SPELL reduced this replication rate to 0.16% without retraining or having to generate-and-discard candidates.
- Minimal Computational Overhead: In intra-batch diversity settings ($K \le 128$), generation time remains virtually indistinguishable from the baseline (e.g., 2.94s vs. 2.93s per image on Simple Diffusion).
Limitations and Future Work
- Shield Overlaps: SPELL’s strict guarantee of landing outside shielded balls mathematically assumes disjoint regions. If shields overlap heavily, exact projections would require solving a quadratic program.
- Metric Space Sensitivity: Currently, distances are computed in RGB space or VAE latent space ($L_2$ norm), capturing low-level visual and color composition. Exploring semantic embeddings (such as DINOv2 feature space) could steer trajectories toward higher-level semantic diversity.
- Scalability Bottlenecks: When repelling from millions of static images, approximate nearest-neighbor retrieval introduces inference latency (e.g., increasing generation time from 2.4s to 4.6–13.5s per image depending on Voronoi probe counts).
Summary updated