TL;DR
The Semantic Color Naturalness Breaker (SCNB) is a proactive, content-side defense mechanism that injects imperceptible perturbations into grayscale images to safeguard them against unauthorized AI colorization. Instead of trying to suppress color entirely, SCNB forces colorization models to produce highly unnatural, content-inconsistent colors (like a bright blue apple or a pink shark), rendering the output useless for copyright infringers while preserving the original grayscale quality for legitimate viewers.
What Problem Does This Paper Address?
Recent advances in generative AI have made automatic image colorization incredibly high-quality and accessible. While great for film restoration, it has also led to widespread unauthorized colorization and redistribution of monochrome media, such as historical photographs and manga panels.
Prior attempts at proactive defense relied on "Uncolorable Examples" (UE)—adding tiny, invisible adversarial perturbations to grayscale images to break downstream colorizers. The baseline method, PAChroma, focused on chroma suppression, attempting to force the AI to output a purely grayscale image.
However, the authors identify a critical flaw in chroma suppression under real-world constraints (like small perturbation budgets or JPEG compression): it often results in a "gray-but-natural" failure mode. The AI still produces a desaturated image, but retains faint, content-consistent tints (e.g., a slightly blue sky), leaving the colorized image visually plausible and still highly usable for bad actors.
Methods
To overcome the limitations of chroma suppression, SCNB shifts the target from color intensity to color plausibility.
1. SCNB Optimization Framework
SCNB treats the defense as an adversarial attack against the colorizer $G(\cdot)$. It optimizes an imperceptible perturbation $\delta$ added to the grayscale image $x_l$ using a perception-aware masked MI-FGSM (Momentum Iterative Fast Gradient Sign Method) backbone. A Laplacian filter is used as an imperceptibility mask to ensure the changes remain completely invisible to human eyes.
2. Content-Aware Color Distributional Distance (CaCDD)
To drive the colorizer toward implausible colors, the authors developed CaCDD, a novel, ground-truth-free metric that acts as both the optimization objective for SCNB and an evaluation metric. CaCDD measures how far a colorized output deviates from semantic color expectations through two components:
- Out-of-Distribution (OoD) Chroma Distance: Measures how far the predicted pixel color falls from a pre-computed cluster of natural colors for that specific object type using Mahalanobis distance.
- Hue-Direction Inconsistency: Penalizes pixels where the hue direction diverges drastically from the expected semantic norm (e.g., turning green vegetation red).
3. Automated Semantic Color Priors
To build the semantic color expectations without manual labeling, the authors designed an automated pipeline. They used the Recognize Anything Model (RAM) for open-vocabulary tagging and Grounded-SAM for pixel-accurate segmentation masks. For each semantic label, they collected color samples in the CIELAB $(a, b)$ space and fitted a $K$-component Gaussian Mixture Model (GMM) to represent the distribution of natural colors for that object.
Datasets
- ImageNet Training Set: Used to automatically extract semantic regions and fit the GMM color priors across 3,032 distinct semantic labels.
- ImageNet Validation Set: 40 images (evaluated across multiple random seeds) were used to benchmark the defense against three state-of-the-art colorizers: DeOldify (CNN-based), BigColor (GAN-based), and DDColor (Transformer-based).
Key Insights and Ideas
- Exploiting Semantic Expectations: The core "aha moment" is that human acceptance of an image depends heavily on semantic context. A desaturated green apple looks fine, but a fully saturated blue apple immediately looks fake. By actively steering the AI toward semantic contradictions rather than trying to zero-out the color, the defense becomes significantly more robust.
- Ground-Truth-Free Evaluation: Traditional metrics like PSNR or SSIM require a single ground-truth colored image. However, colorization is inherently multimodal (a car could be red, blue, or black). CaCDD solves this by evaluating against a statistical distribution of plausible colors for a given label, making it a much more flexible and accurate metric for color naturalness.
Key Results
- Strong Semantic Disruption: SCNB successfully forces target colorizers to yield garish, content-inconsistent outputs. Quantitatively, in white-box settings on BigColor, SCNB increased the CaCDD score from 1.26 to 3.05 (+142%), while PAChroma actually decreased it to 1.05 (confirming the gray-but-natural failure mode).
- High Visual Fidelity: Despite the severe disruption to the AI, the protected grayscale images remain visually indistinguishable from the originals, maintaining high structural similarity (SSIM of ~0.94–0.97).
- Robustness to Post-Processing: SCNB's protection successfully survives common online image degradations, including JPEG compression (at quality levels 75 and 50) and random resized cropping (RRC).
- Black-Box Transferability: When SCNB perturbations are crafted using an ensemble of surrogate colorizers, the defense transfers effectively to completely unseen target architectures.
Limitations and Future Work
- Computational Cost: The primary limitation is optimization time. Generating an uncolorable example takes approximately 10 to 20 minutes per image on an NVIDIA RTX 6000 GPU. While acceptable for a one-time publication cost, it limits real-time deployment.
- Future Directions: The authors plan to extend this framework to video content, adapt it to counter user-hint-conditioned colorization tools, and evaluate its efficacy against stronger, adaptive attackers alongside comprehensive human perception studies.
Summary updated