Suppressing Generative Artifacts via Negative Guidance and Training

18 concepts13 papers1 proposal

Traditional generative and restoration models are typically trained to answer a single question: What should a high-quality image look like? By optimizing objectives that maximize similarity to positive ground-truth examples, models learn to pull their outputs toward a target data distribution.

However, relying solely on positive examples often leaves the boundaries of that distribution ill-defined. Without an explicit understanding of what constitutes a failure, models frequently fall into predictable traps—hallucinating checkerboard patterns, over-smoothing facial features, leaking data augmentations, or producing an "uncanny valley" synthetic appearance.

To resolve this, a powerful paradigm has emerged across image generation and restoration: teaching models what not to generate. By explicitly defining, modeling, and penalizing a "negative space" of artifacts, modern architectures can actively push their generations away from unwanted distortions, resulting in significantly higher fidelity and realism.


Inference-Time Steering via Negative Guidance

The most direct way to utilize negative space is during the sampling process of conditional generative models, such as diffusion models. Through classifier-free guidance, a model can contrast a positive prompt with a negative prompt, effectively calculating a vector field that points away from the negative concept.

For example, in image colorization, models often default to muted or desaturated tones. In Diffusing Colors: Image Colorization with Text Guided Diffusion, Nir Zabari et al. leverage negative prompts like "grainy black-and-white photo" during inference. This forces the diffusion process to actively move away from monochromatic distributions, yielding more vibrant and semantically accurate colorizations.

Standard Guidance:   [Starting Latent] -------> (Positive Prompt: "Vibrant Photo")

Negative Guidance:   [Starting Latent] <------- (Negative Prompt: "Grainy B&W")
                                       \
                                        =======> [Result: High-Fidelity Color]

When an artifact is too complex or subtle to describe in words—such as the distinctively "plastic" texture of 3D-rendered clothing—researchers instead learn the negative space directly. In FashionR2R: Texture-preserving Rendered-to-Real Image Translation with Diffusion Models, Chen Huang et al. use textual inversion to capture the representational embedding of synthetic rendering artifacts. By applying this learned embedding as negative guidance, the model explicitly suppresses the uncanny valley effects of computer-generated skin and fabric without losing the underlying asset geometry.

This strategy is equally effective for correcting hardware-level distortions. In Degradation-Modeled Multipath Diffusion for Tunable Metalens Photography, the authors use a negative prompt to learn and mimic complex optical aberrations and sensor noise unique to ultra-compact metalenses. During reconstruction, the model uses this learned profile to exclude those specific degradations from the final output.

Furthermore, this negative steering capability can be compressed into faster architectures. Ruiqing Wang et al. demonstrate in MFSR: MeanFlow Distillation for One Step Real-World Image Super Resolution that incorporating negative prompts into a teacher model's guidance during distillation successfully transfers artifact-suppression capabilities to a fast, single-step student model.


Reshaping the Loss Landscape with Negative Data Augmentation

While inference-time guidance steers a pre-trained model, incorporating negative samples directly into training reshapes the model's underlying loss landscape. This is particularly vital in Generative Adversarial Networks (GANs), where data scarcity often forces practitioners to use heavy data augmentations that accidentally "leak" into the generated images.

To prevent a generator from mimicking these training distortions, Zhaoyu Zhang et al. introduce Adaptive Negative Data Augmentation (ANDA) in Improving the Leaking of Augmentations in Data-Efficient GANs via Adaptive Negative Data Augmentation. Instead of only training the discriminator on clean real images and synthetic images, the authors feed heavily augmented real images to the discriminator and label them explicitly as fake. This creates an explicit penalty for the generator if its outputs resemble the augmented distribution, guiding it strictly toward the clean, underlying data distribution.

Discriminator Training:

- Clean Real Images       --> Label: REAL
  - Generated Images        --> Label: FAKE
  - Augmented Real Images   --> Label: FAKE (Negative Anchor)

A similar philosophy is explored by Xin Ding et al. in Turning Waste into Wealth: Leveraging Low-Quality Samples for Enhancing Continuous Conditional Generative Adversarial Networks. Their framework, Dual-NDA, treats low-quality synthetic samples as negative training data. By explicitly defining what constitutes a failed generation, the model learns to avoid common distortion patterns, pushing the generator toward higher visual sharpness.

This adversarial rejection of negative spaces can even be applied to post-training modification. In Data Redaction from Pre-trained GANs, Zhifeng Kong et al. propose an exemplar-based suppression method. By treating a predefined set of undesirable outputs as additional negative examples for the discriminator during fine-tuning, they update the generator's gradients to push its output distribution away from those specific points, effectively redacting data without a costly full retrain.


Hard Negatives and Quality Anchors in Image Restoration

In image restoration tasks like super-resolution, denoising, and dehazing, models often introduce subtle, high-frequency artifacts that standard pixel-wise losses (like L1 or L2) fail to penalize adequately. To fix this, recent frameworks introduce "hard negatives"—flawed images that are close to the target distribution but contain specific, undesirable defects.

In Perceive-IR: Learning to Perceive Degradation Better for All-in-One Image Restoration, Xu Zhang et al. utilize the outputs of other state-of-the-art restoration models as hard negatives during training. This difficulty-adaptive approach forces the network to distinguish its own results from high-quality but subtly flawed proxy outputs, teaching it to identify and avoid common model-generated distortions.

Similarly, Jiangshe Zhang et al. employ a contrastive refinement process in Spherical Space Feature Decomposition for Guided Depth Map Super-Resolution. They generate synthetic negative samples that represent modality-specific errors, such as texture leakage from an RGB guide into a depth map. By pushing the model's predictions away from these synthetic failure modes while pulling them toward the ground truth, the network learns to actively suppress noise.

To ground these negative spaces in actual human perception, Du Chen et al. developed the HGGT dataset in Human Guided Ground-truth Generation for Realistic Image Super-resolution. Instead of assuming original high-resolution images are perfect, human subjects curated pairs of positive patches and negative patches containing over-sharpening or checkerboard patterns. Training with these explicit negative samples provides a critical corrective signal that prevents the model from replicating known generative errors.

Contrastive Restoration Alignment:
  [Anchor: Low-Quality Input] 
         |
         +---> Pull toward: [Positive Ground-Truth (Human Approved)]
         |
         +---> Push away from: [Hard Negative (Over-sharpened / Artifact-ridden)]

Aligning models with human preferences can also be achieved by mapping these artifacts to semantic anchors during training. In Human Preference Score: Better Aligning Text-to-image Models with Human Preference, Xiaoshi Wu et al. show that associating undesirable image characteristics (like mangled limbs or poor compositions) with a specific negative textual identifier allows a model to isolate and categorize these features, making them easy to suppress.

This paradigm scales seamlessly to large foundation models. In Scaling Up to Excellence: Practicing Model Scaling for Photo-Realistic Image Restoration In the Wild, Fanghua Yu et al. incorporate low-quality samples directly into the training set of a 2.6-billion-parameter SDXL-based model (SUPIR). This allows the model to learn a robust internal representation of blur and distortion, which users can dynamically push against using negative prompts at inference time.

A similar dual-control strategy is utilized by Keda TAO et al. in Overcoming False Illusions in Real-World Face Restoration with Multi-Modal Guided Diffusion Model, where negative quality samples and descriptive prompts are used during training to leverage classifier-free guidance against over-smoothing and artificial textures.

Finally, instead of relying solely on text prompts, this negative push can be driven by automated metrics. In IQA-Adapter: Exploring Knowledge Transfer from Image Quality Assessment to Diffusion-based Generative Models, the authors integrate pre-trained Image Quality Assessment (IQA) models into the generation loop. By actively pushing the diffusion process away from states that the IQA model identifies as 'bad' quality, the adapter ensures technically sound and aesthetically pleasing outputs without sacrificing generative diversity.

Key Foundational Work

Human Guided Ground-truth Generation for Realistic Image Super-resolution (2023) — contributed 2 concept elements to this direction.
Improving the Leaking of Augmentations in Data-Efficient GANs via Adaptive Negative Data Augmentation (2024) — contributed 2 concept elements to this direction.
Spherical Space Feature Decomposition for Guided Depth Map Super-Resolution (2023) — contributed 2 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_artifact_suppression_e78c2cbe,
  title = {artifact suppression},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/artifact-suppression/rdr_e78c2cbefc504d58b263bd4e1705641a},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_artifact_suppression_e78c2cbe_v1,
  title = {artifact suppression},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/artifact-suppression/rdr_e78c2cbefc504d58b263bd4e1705641a/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake