TL;DR
Coupled Spectral Perturbation (CosPert) is a training-free, "plug-and-play" method that boosts the creativity of text-to-image (T2I) models by strategically perturbing their internal weights at inference time. By targeting specific attention layers and using a time-decaying schedule, CosPert breaks rigid visual conventions to generate novel object structures while maintaining high image quality and prompt alignment.
What Problem Does This Paper Address?
Modern T2I models like FLUX.1 and Stable Diffusion are victims of their own success: because they are trained on massive datasets of existing images, they tend to favor "familiar visual conventions." When a user asks for a "creative chair," the model often produces a standard chair with minor decorative flourishes rather than a truly novel form.
Existing solutions usually try to inject creativity by modifying the text prompt or manipulating intermediate features during the generation process. However, the model's "preference" for the ordinary is fundamentally baked into its pretrained weights. Simply adding random noise to these weights usually breaks the model, leading to artifacts or loss of semantic meaning. The authors set out to find a way to perturb these weights structurally to unlock creative variations without destroying the model's ability to generate coherent images.
Methods
CosPert introduces a three-step approach to "creative weight-space exploration":
- Layer-wise Sensitivity Analysis: Not all parts of a neural network are equally important for creativity. The authors perform a one-time analysis to identify which layers' weights most significantly impact the output when changed. They found that perturbing just a few key attention projection layers is enough to trigger major structural changes.
- Coupled Spectral Perturbation: Instead of adding unstructured noise, CosPert uses Singular Value Decomposition (SVD) to decompose weight matrices into their "spectral" components (singular values and vectors). It then applies a shared random transformation that "couples" updates to both input and output directions while perturbing the spectral gains. This structured approach ensures the weights remain "meaningful" to the network.
- Sigmoid Decay Schedule: Timing is everything. The authors observed that the global structure of an image is determined early in the diffusion process, while fine details are refined at the end. CosPert applies the strongest weight perturbations at the start of denoising and gradually fades them out using a sigmoid schedule. This allows the model to "dream" up a novel structure early on and then use its original, unperturbed weights to "polish" the final image.
Datasets
The authors evaluated their method using a benchmark of 50 common object categories, ranging from manufactured items (e.g., "Bed," "Piano," "Vase") to natural objects (e.g., "Marine Animal," "Vegetable"). They compared CosPert against several state-of-the-art baselines, including GPT-Image 2.5, Nano Banana 2.1, and specialized creativity methods like C3 and SCDiff.
Key Insights and Ideas
- Weights are a Creative Goldmine: While most research focuses on prompts or features, the pretrained weights themselves offer a complementary and powerful source of variation that has been largely overlooked.
- Structure Over Randomness: Random weight jittering is destructive. By perturbing weights in their spectral (SVD) domain, you can shift the model's "internal concepts" without breaking its fundamental logic.
- The "Aha" Moment of Timing: You don't need to change the model forever. By only perturbing weights during the "formative" early steps of image generation, you get the best of both worlds: radical new shapes and high-fidelity textures.
- Sensitivity Matters: As shown in Figure 2, the model's response to weight changes is highly non-uniform. Targeting the "most responsive" layers (often at the beginning and end of the backbone) is crucial for efficiency and effect.
Key Results
- Creativity Metrics: CosPert achieved the highest mean CREward (a metric specifically designed to measure visual novelty in geometry and texture) across 50 categories, significantly outperforming standard FLUX.1 and other creativity-focused methods.
- Alignment and Preference: It maintained strong text-image alignment (measured by VQAScore) and received the highest preference scores in a user study, beating out even LLM-guided creative agents.
- Efficiency: Because it is a "plug-and-play" weight modification, CosPert requires zero additional training and adds negligible overhead to the inference time. Once the perturbations are calculated (a one-time cost), sampling is as fast as the original model.
Limitations and Future Work
The authors note that while CosPert is training-free, the sensitivity analysis must be performed once for each new model architecture (e.g., once for FLUX.1, once for SD3.5).
Future work could explore more complex "coupling" strategies or investigate if these weight-space perturbations can be conditioned on specific artistic styles rather than just being stochastic. There is also potential to combine CosPert with existing prompt-based creativity methods for even more radical results.
Summary updated