Breaking Visual Conventions: Spectrally Coupled Weight Perturbation for Creative Image Generation

Fu Feng, Ruoheng Zhou, Yucheng Xie, Xu Yang, Jing Wang, Xin GengarXiv 2026 · arXiv

TL;DR

Coupled Spectral Perturbation (CosPert) is a training-free, "plug-and-play" method that boosts the creativity of text-to-image (T2I) models by strategically perturbing their internal weights at inference time. By targeting specific attention layers and using a time-decaying schedule, CosPert breaks rigid visual conventions to generate novel object structures while maintaining high image quality and prompt alignment.

What Problem Does This Paper Address?

Modern T2I models like FLUX.1 and Stable Diffusion are victims of their own success: because they are trained on massive datasets of existing images, they tend to favor "familiar visual conventions." When a user asks for a "creative chair," the model often produces a standard chair with minor decorative flourishes rather than a truly novel form.

Existing solutions usually try to inject creativity by modifying the text prompt or manipulating intermediate features during the generation process. However, the model's "preference" for the ordinary is fundamentally baked into its pretrained weights. Simply adding random noise to these weights usually breaks the model, leading to artifacts or loss of semantic meaning. The authors set out to find a way to perturb these weights structurally to unlock creative variations without destroying the model's ability to generate coherent images.

Methods

CosPert introduces a three-step approach to "creative weight-space exploration":

  1. Layer-wise Sensitivity Analysis: Not all parts of a neural network are equally important for creativity. The authors perform a one-time analysis to identify which layers' weights most significantly impact the output when changed. They found that perturbing just a few key attention projection layers is enough to trigger major structural changes.
  2. Coupled Spectral Perturbation: Instead of adding unstructured noise, CosPert uses Singular Value Decomposition (SVD) to decompose weight matrices into their "spectral" components (singular values and vectors). It then applies a shared random transformation that "couples" updates to both input and output directions while perturbing the spectral gains. This structured approach ensures the weights remain "meaningful" to the network.
  3. Sigmoid Decay Schedule: Timing is everything. The authors observed that the global structure of an image is determined early in the diffusion process, while fine details are refined at the end. CosPert applies the strongest weight perturbations at the start of denoising and gradually fades them out using a sigmoid schedule. This allows the model to "dream" up a novel structure early on and then use its original, unperturbed weights to "polish" the final image.
Figure 1: CosPert generates unconventional forms like rose-petal skirts and geometric lamps while preserving the core category identity.

Datasets

The authors evaluated their method using a benchmark of 50 common object categories, ranging from manufactured items (e.g., "Bed," "Piano," "Vase") to natural objects (e.g., "Marine Animal," "Vegetable"). They compared CosPert against several state-of-the-art baselines, including GPT-Image 2.5, Nano Banana 2.1, and specialized creativity methods like C3 and SCDiff.

Key Insights and Ideas

  • Weights are a Creative Goldmine: While most research focuses on prompts or features, the pretrained weights themselves offer a complementary and powerful source of variation that has been largely overlooked.
  • Structure Over Randomness: Random weight jittering is destructive. By perturbing weights in their spectral (SVD) domain, you can shift the model's "internal concepts" without breaking its fundamental logic.
  • The "Aha" Moment of Timing: You don't need to change the model forever. By only perturbing weights during the "formative" early steps of image generation, you get the best of both worlds: radical new shapes and high-fidelity textures.
  • Sensitivity Matters: As shown in Figure 2, the model's response to weight changes is highly non-uniform. Targeting the "most responsive" layers (often at the beginning and end of the backbone) is crucial for efficiency and effect.
Figure 2. Layer Sensitivity to Weight Perturbation in FLUX.1. (a) We apply Gaussian perturbations one layer at a time at relative magnitudes ρ = 3.0 and ρ = 5.0, with ρ defined in Eq. (8). The images illustrate how the effects vary with the target layer and perturbation strength. (b) We rank layers by the normalized change in the backbone prediction and apply CosPert to the five highest-scoring layers.

Key Results

  • Creativity Metrics: CosPert achieved the highest mean CREward (a metric specifically designed to measure visual novelty in geometry and texture) across 50 categories, significantly outperforming standard FLUX.1 and other creativity-focused methods.
  • Alignment and Preference: It maintained strong text-image alignment (measured by VQAScore) and received the highest preference scores in a user study, beating out even LLM-guided creative agents.
  • Efficiency: Because it is a "plug-and-play" weight modification, CosPert requires zero additional training and adds negligible overhead to the inference time. Once the perturbations are calculated (a one-time cost), sampling is as fast as the original model.

Limitations and Future Work

The authors note that while CosPert is training-free, the sensitivity analysis must be performed once for each new model architecture (e.g., once for FLUX.1, once for SD3.5).

Future work could explore more complex "coupling" strategies or investigate if these weight-space perturbations can be conditioned on specific artistic styles rather than just being stochastic. There is also potential to combine CosPert with existing prompt-based creativity methods for even more radical results.

Summary updated

Abstract

Text-to-image (T2I) models often produce high-quality images that follow familiar visual conventions, even in response to open-ended prompts such as ``a creative [CLS]''. Existing methods encourage novelty by exploring semantic conditions or modifying features and guidance during denoising. Pretrained weights offer a complementary source of creative variation, but unstructured perturbations can have little visual effect or degrade image quality. We introduce Coupled Spectral Perturbation (CosPert), a training-free method that targets attention projections in layers identified by a one-time sensitivity analysis. For each projection, CosPert uses a shared random transformation and its transpose to couple updates to the singular-vector factors while perturbing the spectral gains. A common relative budget calibrates the magnitude of weight changes across projections, and a sigmoid schedule concentrates their influence in early denoising and gradually attenuates them during refinement. CosPert provides plug-and-play creativity enhancement at inference time. Once the perturbations are prepared, sampling requires no auxiliary networks or additional model evaluations. On both FLUX.1 and Stable Diffusion 3.5, CosPert enhances visual creativity, producing more varied object structures while preserving category identity. Across 50 object categories, CosPert achieves the highest mean CREward among the evaluated methods while maintaining strong prompt alignment.
BibTeX
@misc{feng2026breaking,
  author = {Fu Feng and Ruoheng Zhou and Yucheng Xie and Xu Yang and Jing Wang and Xin Geng},
  title = {{Breaking Visual Conventions: Spectrally Coupled Weight Perturbation for Creative Image Generation}},
  year = {2026},
  eprint = {2611.22318},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.22318},
}
Figure