Fine-Grained Safety Taxonomies as an Attack Surface for Multimodal Guardrails

Deployed vision-language systems increasingly put a safety classifier in front of the generator. This classifier, called a guardrail, reads the image and the prompt and assigns them to categories from a risk taxonomy. Each category then triggers its own action: refuse, warn, or allow. The taxonomies keep growing, because finer categories let operators write finer policies. USB, for example, uses 61 tertiary risk categories. Work on multi-label classifiers points to a cost here. The multi-label setting is the one where a model assigns several non-exclusive labels to a single input. In that setting, more labels make a model easier to fool, not harder, because every additional label gives a gradient-based attacker another "distractor" it can promote in place of the true one. We propose to test whether this effect carries over to multimodal safety guardrails. The specific worry is an imperceptible image perturbation that does not suppress a hazard but relabels it into a sibling category whose policy is permissive, such as moving "weapon synthesis" into "fictional violence."

If the effect is real, it changes how guardrails should be built and defended. Collapsing everything to a binary safe/unsafe decision removes the attack, but it also removes the policy granularity that motivated the taxonomy. Full adversarial training of a billion-parameter vision encoder is too expensive for most teams, and it typically costs clean accuracy, which for a guardrail means refusing harmless requests. We will measure the vulnerability as a function of taxonomy size. We will then test a cheap defense: a handful of learnable input tokens added to a frozen encoder. These tokens are trained against relabeling attacks, and the training weights each possible relabeling by how much policy it would bypass. The goal is to keep the fine taxonomy and its clean behavior while closing the cross-category escape routes.

Core Claim

The claim has three parts. First, multi-label safety guardrails over multimodal inputs show the label-cardinality effect: the smallest image perturbation that evades the policy shrinks as the taxonomy is refined, holding backbone and training data fixed. Second, the dominant evasion mode under category-dependent policies is targeted relabeling into a permissive category, not wholesale suppression to "safe." Third, this mode can be largely closed without full adversarial training and without raising false refusals. The defense trains a few robustness tokens on a frozen encoder against a ranking-aware relabeling attack, with each relabeling weighted by its policy cost.

The first two parts are diagnostic, and they could fail. If evasion budgets do not shrink with taxonomy size, the distractor mechanism does not transfer to safety guardrails, and the defense question reduces to ordinary binary robustness.

Each strand of prior work plays a different role here. TkML-AP motivates the claim. It builds a convex relaxation of top-k ranking and uses it to craft untargeted, targeted, and universal attacks on multi-label image taggers, and it links attack success to label cardinality. It studies object tags on PASCAL VOC and COCO, not safety categories tied to actions, and it proposes no defense. Characterizing the Evasion Attackability of Multi-label Classifiers supplies the measurement idea. It defines attackability as the maximum number of labels flippable within a budget, approximates it with a greedy submodular search (GASE), and regularizes toward low-rank weights. We reuse its attackability notion, but we replace raw label-flip counts with policy cost, and we work on modern frozen encoders, where nuclear-norm regularization over full weights is impractical. Multi-label Adversarial Perturbations shows that ranking-targeted attacks (its "Rank II") can flip a chosen label while leaving the others intact. That is exactly the selective relabeling we expect against guardrails.

Research Program

The guardrail we study is a frozen vision-language encoder. It is CLIP-style, so the vision and text towers can be perturbed and probed separately. A lightweight multi-label head on top of it is trained on USB-Base (USB; data at cgjacklin/USB). USB's three-tier hierarchy (3, 16, and 61 categories) gives us four taxonomy granularities from the same data: binary, coarse, mid, and fine. Each USB sample has a tertiary path, so each sample carries its ancestor labels.

To create genuinely multi-hazard inputs, we pair a risky image from one category with risky text from another. We also keep USB's Safe Image/Safe Text cases. We add SIUO as a held-out test of compositional risk, meaning inputs that are harmless in each modality alone and harmful only in combination. We write a policy matrix that maps each fine category to refuse, warn, or allow. "Policy cost" is the severity gap when an input whose true action is refuse is pushed into a category with a laxer action.

The attack suite has three members. The first is TkML-AP-style untargeted suppression of all hazard labels. The second is targeted relabeling into the most permissive reachable category; this adapts TkML-AP-T and Rank II to the policy matrix. The third is a GASE-style greedy search for the largest policy cost achievable under a budget. Perturbations are on image pixels at ℓ∞ budgets of 1/255 through 8/255, matching the regime in which On the Adversarial Robustness of Multi-Modal Foundation Models hijacked OpenFlamingo. All attacks are adaptive: they see the defense's tokens and differentiate through them. This matters because the robustness-token paper itself flags token secrecy as a weakness.

The defense reuses the mechanism of Robustness Tokens. That paper appends about ten learnable tokens to a frozen ViT and optimizes them in a few hundred steps with two losses: one keeping clean features unchanged and one correcting features under attack. We change what "correction" means. Instead of restoring features uniformly, the adversarial term is the head's ranking loss on perturbed inputs produced by the relabeling attack, weighted by the policy cost of the relabeling found. Let $c(y, \hat{y})$ denote the policy-cost of moving from true label set $y$ to predicted set $\hat{y}$. Let $\delta^*$ denote the cost-weighted attack's perturbation, and let $z$ and $z_\tau$ denote encoder features without and with tokens $\tau$. The token objective is then

$$\min_\tau \; \mathbb{E}\big[\, \|z_\tau(x) - z(x)\|^2 \;+\; \lambda\, c\big(y, \hat{y}_\tau(x+\delta^*)\big)\, \ell_{\text{rank}}\big(x+\delta^*, y\big) \big],$$

where $\ell_{\text{rank}}$ is the average-of-top-k ranking loss from TkML-AP and $\lambda$ trades clean invariance against correction. The first term is what should keep clean refusal behavior intact. The cost weight is the ingredient we claim matters. It spends the small token budget on the cross-category boundaries that the policy actually cares about.

Baselines

The naive reference point is the undefended fine-taxonomy guardrail. It sets the attackability curves that every defense must improve. Next to it is the binary-collapse guardrail: the same backbone and data, with all hazards merged into one label. Binary collapse is the cheapest structural fix, so it bounds how much robustness is available from simply discarding granularity.

The strongest existing parameter-efficient competitor is vanilla Robustness Tokens: ten tokens trained with the original uniform feature-correction objective. It is the closest method in compute footprint, and it is the only one in the file reported to keep clean accuracy at frozen-backbone cost. On DINOv2-Giant, ImageNet accuracy under white-box attack went from 0.3% to 63.1%, while clean accuracy stayed at 85.2% versus 85.6% (Robustness Tokens). There is no reported result on Lacuna for robustness tokens, or for any defense, on a multi-label safety guardrail. We will run it ourselves with the same token count, step budget, learning-rate sweep, and attack budget as our method.

The strongest compute-unconstrained competitor is multi-label PGD adversarial training of the head plus last encoder blocks, using a per-label binary cross-entropy inner maximization. We give it ten times our GPU-hours so that it serves as a ceiling, not a strawman. We also include VARMAT, a single-step gradient-norm regularizer applied to whichever modality its first-order probe flags as fragile. It reported absolute robustness gains of 12.73%, 22.21%, and 11.19% on CMU-MOSEI, UR-FUNNY, and AVMNIST. Those are single-label tasks, so we run it on the guardrail at matched compute.

The key ablation swaps the cost-weighted relabeling attack in our objective for uniform label-flip attacks. This isolates whether policy weighting, rather than tokens per se, drives the gains. Beating the strongest baseline means two things together: at equal budget, lower policy-cost evasion than vanilla tokens and than adversarial training at its larger compute, and a false-refusal rate on benign USB inputs no more than one point above the undefended guardrail.

Why not just…?

Why not just collapse the taxonomy to binary safe/unsafe? This is the most serious threat, and the scaling experiment addresses it directly. If the binary guardrail turns out as robust as our defended fine guardrail, and nearly as accurate, then the right recommendation is "decide coarsely, label finely afterward," and the token method is unnecessary. We therefore report the binary model's evasion budget and its loss of policy expressiveness, meaning the fraction of warn-or-allow cases it is forced to refuse. The proposal only stands if fine taxonomies defended with tokens dominate the binary model on that joint frontier. The pattern in USB suggests blunt decisions are costly. Claude 3.5 Sonnet had the lowest attack success (15.57%) but refused 25.82% of harmless prompts. That is suggestive, not proof.

Why not just purify inputs, for example with diffusion denoising or JPEG? Multi-label Adversarial Perturbations found JPEG restores labels against basic attacks but is "not a robust solution" against adaptive ones. Purification also cannot help with SIUO-style compositional cases, which contain no adversarial noise at all. We include DiffPure-RobustVLM under adaptive attack, and we report its added latency and its benign false-refusal rate.

Why not just tell the model in its prompt that images may be adversarial? Improving Adversarial Robustness in Vision-Language Models with Architecture and Prompt Design reports gains from exactly this at near-zero cost, so it is a fair zero-compute baseline: we prepend the warning to the text tower. We expect it to be weak against pixel attacks that corrupt features before any text is read, as in the OpenFlamingo hijacking. That is an expectation, and the experiment decides it.

Why not just steer the downstream generator's refusal direction instead of hardening the guardrail? SRT lowered Qwen3-VL-8B's SIUO unsafe rate from 46.7% to 19.76% by steering hidden states toward a refusal direction. That is complementary, not competing. SRT makes a scalar refuse/comply decision, so it has no notion of category-dependent policy, and it was not tested against white-box pixel attacks. We run SRT behind each guardrail and ask whether guardrail relabeling attacks still pass end to end.

Proposed Experiments and Decisive Checks

The first decisive check is the scaling curve. For each granularity (1, 3, 16, 61 categories) we measure the median minimum ℓ∞ budget for policy evasion and the GASE-style maximum policy cost at a fixed budget, across three backbones. The hypothesis fails if the budget does not fall monotonically with granularity in at least two of the three backbones. We also report which attack mode dominates; we predict targeted relabeling beats suppression, which follows from the Rank II result that selective flips are cheap.

The second check is the defense comparison above, run on USB-Base and USB-Hard. It reports policy-cost-weighted evasion rate, top-k recall of true hazard labels under attack, benign false-refusal rate on Safe Image/Safe Text, and GPU-hours. The cost-weighted versus uniform ablation decides whether our ingredient matters.

The third check is transfer. We test tokens trained on USB against SIUO compositional cases, against universal perturbations, and end to end in front of a LLaVA-class generator. For the end-to-end test, harmfulness is judged by the HarmBench classifier, which reports 93.19% agreement with humans.

A clean negative result is also publishable. If the curve is flat, or binary collapse matches the defended fine guardrail, guardrail designers learn that taxonomy refinement is free, or that it should be separated from the refusal decision.