Semantic Color Naturalness Breaker: Preventing Illegitimate Colorization via Content-Aware Color Priors

Yuki Nii, Futa Waseda, Ching-Chun Chang, Isao EchizenarXiv 2026 · arXiv

TL;DR

The Semantic Color Naturalness Breaker (SCNB) is a proactive, content-side defense mechanism that injects imperceptible perturbations into grayscale images to safeguard them against unauthorized AI colorization. Instead of trying to suppress color entirely, SCNB forces colorization models to produce highly unnatural, content-inconsistent colors (like a bright blue apple or a pink shark), rendering the output useless for copyright infringers while preserving the original grayscale quality for legitimate viewers.

What Problem Does This Paper Address?

Recent advances in generative AI have made automatic image colorization incredibly high-quality and accessible. While great for film restoration, it has also led to widespread unauthorized colorization and redistribution of monochrome media, such as historical photographs and manga panels.

Prior attempts at proactive defense relied on "Uncolorable Examples" (UE)—adding tiny, invisible adversarial perturbations to grayscale images to break downstream colorizers. The baseline method, PAChroma, focused on chroma suppression, attempting to force the AI to output a purely grayscale image.

However, the authors identify a critical flaw in chroma suppression under real-world constraints (like small perturbation budgets or JPEG compression): it often results in a "gray-but-natural" failure mode. The AI still produces a desaturated image, but retains faint, content-consistent tints (e.g., a slightly blue sky), leaving the colorized image visually plausible and still highly usable for bad actors.

Fig. 1: SCNB breaks semantic color plausibility. UE generated from PAChroma suppresses chroma magnitude but can leave gray-but-natural outputs, whereas SCNB induces content-inconsistent colors even under restricted settings.

Methods

To overcome the limitations of chroma suppression, SCNB shifts the target from color intensity to color plausibility.

1. SCNB Optimization Framework

SCNB treats the defense as an adversarial attack against the colorizer $G(\cdot)$. It optimizes an imperceptible perturbation $\delta$ added to the grayscale image $x_l$ using a perception-aware masked MI-FGSM (Momentum Iterative Fast Gradient Sign Method) backbone. A Laplacian filter is used as an imperceptibility mask to ensure the changes remain completely invisible to human eyes.

2. Content-Aware Color Distributional Distance (CaCDD)

To drive the colorizer toward implausible colors, the authors developed CaCDD, a novel, ground-truth-free metric that acts as both the optimization objective for SCNB and an evaluation metric. CaCDD measures how far a colorized output deviates from semantic color expectations through two components:

  • Out-of-Distribution (OoD) Chroma Distance: Measures how far the predicted pixel color falls from a pre-computed cluster of natural colors for that specific object type using Mahalanobis distance.
  • Hue-Direction Inconsistency: Penalizes pixels where the hue direction diverges drastically from the expected semantic norm (e.g., turning green vegetation red).

3. Automated Semantic Color Priors

To build the semantic color expectations without manual labeling, the authors designed an automated pipeline. They used the Recognize Anything Model (RAM) for open-vocabulary tagging and Grounded-SAM for pixel-accurate segmentation masks. For each semantic label, they collected color samples in the CIELAB $(a, b)$ space and fitted a $K$-component Gaussian Mixture Model (GMM) to represent the distribution of natural colors for that object.

Datasets

  • ImageNet Training Set: Used to automatically extract semantic regions and fit the GMM color priors across 3,032 distinct semantic labels.
  • ImageNet Validation Set: 40 images (evaluated across multiple random seeds) were used to benchmark the defense against three state-of-the-art colorizers: DeOldify (CNN-based), BigColor (GAN-based), and DDColor (Transformer-based).

Key Insights and Ideas

  • Exploiting Semantic Expectations: The core "aha moment" is that human acceptance of an image depends heavily on semantic context. A desaturated green apple looks fine, but a fully saturated blue apple immediately looks fake. By actively steering the AI toward semantic contradictions rather than trying to zero-out the color, the defense becomes significantly more robust.
  • Ground-Truth-Free Evaluation: Traditional metrics like PSNR or SSIM require a single ground-truth colored image. However, colorization is inherently multimodal (a car could be red, blue, or black). CaCDD solves this by evaluating against a statistical distribution of plausible colors for a given label, making it a much more flexible and accurate metric for color naturalness.
Fig. 4: CaCDD captures content-aware color plausibility. Distinguishing natural from semantically implausible col- orizations beyond CF/PSNR/SSIM.

Key Results

  • Strong Semantic Disruption: SCNB successfully forces target colorizers to yield garish, content-inconsistent outputs. Quantitatively, in white-box settings on BigColor, SCNB increased the CaCDD score from 1.26 to 3.05 (+142%), while PAChroma actually decreased it to 1.05 (confirming the gray-but-natural failure mode).
  • High Visual Fidelity: Despite the severe disruption to the AI, the protected grayscale images remain visually indistinguishable from the originals, maintaining high structural similarity (SSIM of ~0.94–0.97).
  • Robustness to Post-Processing: SCNB's protection successfully survives common online image degradations, including JPEG compression (at quality levels 75 and 50) and random resized cropping (RRC).
  • Black-Box Transferability: When SCNB perturbations are crafted using an ensemble of surrogate colorizers, the defense transfers effectively to completely unseen target architectures.
Fig. 2: Uncolorable Example generated from SCNB inducing implausible color. Each image is shown with its SSIM between the inputs and CaCDD score.

Limitations and Future Work

  • Computational Cost: The primary limitation is optimization time. Generating an uncolorable example takes approximately 10 to 20 minutes per image on an NVIDIA RTX 6000 GPU. While acceptable for a one-time publication cost, it limits real-time deployment.
  • Future Directions: The authors plan to extend this framework to video content, adapt it to counter user-hint-conditioned colorization tools, and evaluate its efficacy against stronger, adaptive attackers alongside comprehensive human perception studies.

Summary updated

Abstract

Automatic image colorization enables large-scale and low-cost reuse of grayscale media (e.g., manga panels and archival photographs), facilitating unauthorized reuse and redistribution. Once released online, grayscale content can be readily turned into unauthorized colorized derivatives using off-the-shelf models, creating a practical need for proactive, content-side protection at publication time. Building on Uncolorable Examples (UE), which add imperceptible perturbations to released grayscale images to degrade unauthorized colorization, we propose Semantic Color Naturalness Breaker (SCNB) -- a semantic-level UE framework that drives colorization outputs toward content-inconsistent colors while preserving the visual fidelity of the released grayscale media. We further introduce Content-aware Color Distributional Distance (CaCDD), a ground-truth-free, content-aware measure of color plausibility derived from semantic color priors, used both as the optimization objective of SCNB and as an evaluation metric. Experiments on ImageNet show that our method remains effective under small perturbation budgets and common post-processing, supporting practical deployment in real-world content-sharing pipelines.
BibTeX
@misc{nii2026semantic,
  author = {Yuki Nii and Futa Waseda and Ching-Chun Chang and Isao Echizen},
  title = {{Semantic Color Naturalness Breaker: Preventing Illegitimate Colorization via Content-Aware Color Priors}},
  year = {2026},
  eprint = {2607.17610},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2607.17610},
}
Figure