Object-Graph Consistency Losses for Fast Adversarial Patch Inpainting

An adversarial patch is a small, high-contrast sticker or digital overlay that makes an object detector see things that are not there or miss things that are. The most effective defenses work in two steps. They find the patch, then fill the hole with plausible pixels. Filling matters because a black box left in its place gets detected as a "TV" or "suitcase", or makes the real object vanish. The fastest filling method in use is coherence transport, a classical algorithm that pushes surrounding pixel values into the hole along the direction of local edges. It is cheap enough for edge devices, but it is blind to objects. It smears background across an object's boundary, cuts off a pedestrian's occluded leg, and sometimes paints a shape the detector reads as a new object. Diffusion inpainters produce better fills but take many denoising passes, which is too slow for real-time perception.

We propose to train a small corrective network on top of coherence transport. Its training loss is defined over objects rather than pixels: a frozen detector parses the clean image into an object graph (detected objects as nodes, their pairwise layout as edges), and the inpainted image is penalized whenever that graph changes. Objects that cross the hole must keep their class and extent. Their spatial relations to neighbors must survive. Nothing new may appear inside the hole. We call the inpainter conservative: it tries to leave the scene's object structure exactly as the uncorrupted context implies, without guessing what the patch hid. The bet is that most of the remaining gap between fast and slow inpainters comes from boundary damage and ghost detections, not from texture realism, and that a structured loss targets that gap directly.

Core Claim

A lightweight coherence-transport-plus-residual inpainter, trained with a loss that measures whether a frozen detector's object graph is preserved, will match diffusion inpainting on detection accuracy under patch attack at a small fraction of the latency. We expect the advantage to show up in two places. The first is cases where the patch partially overlaps a real object. The second is the rate of false detections created by the fill. We do not claim recovery when a patch fully covers an object. Following the information-loss argument in On The Classification-Distortion-Perception Tradeoff, which shows that post-processing cannot add task information absent from the observed signal, we treat confident reconstruction of fully hidden content as a hallucination risk to be measured, not a goal.

This differs from the closest work in specific ways. Defending Adversarial Patches via Joint Region Localizing and Inpainting (RLID) trains its inpainter with an L1, adversarial, and VGG-19 feature loss. None of these terms knows about objects, and RLID was evaluated mostly on traffic-sign crops. PAD: Patch-Agnostic Defense against Adversarial Patch Attacks localizes patches well but treats inpainting as an uncoupled black box. LOGICDEF uses scene-graph context to correct a classifier's output logits, but it never touches pixels, so the patch remains in the image. Image Generation From Scene Graphs conditions a generator on a graph to synthesize whole images. We neither generate from a graph nor condition on one. The graph appears only inside the training loss and is discarded at test time.

Research Program

The inpainter has three parts. First, coherence transport, as used in An information-theoretic perspective of physical adversarial patches, produces an initial fill. Second, a small encoder-decoder (a few hundred thousand to a few million parameters) takes the original image, the mask, and the coherence-transport fill, and predicts a residual correction restricted to a band around the hole.

Third, the network's skip connections carry high-resolution features from the encoder straight to the decoder, which is how encoder-decoders keep fine detail. Here they are gated by a soft mask. LaDI-VTON introduced hard mask-gated skips (EMASC) for virtual try-on, where masks are clean. Patch masks from real localizers are ragged, and Segment and Complete (SAC) notes that leaving even a sliver of patch unmasked lets adversarial signal spread through a deep network. Our gate is therefore a distance-transform ramp that is fully closed within a margin of the predicted mask and opens smoothly outside it. It is trained with masks that are randomly dilated, eroded, and fragmented to mimic localizer errors. The ramp trades leakage against the seams that hard gating creates.

Training images are clean COCO images with digitally pasted patches, so the clean image $x$ is always available. A frozen detector $D$ (Faster R-CNN) is run on $x$ to produce region proposals and detections. The object graph has these detections as nodes and their pairwise relative box offsets as edges. For the inpainted image $\hat{x}$ we re-run the detector's region head on the clean image's own regions, which keeps everything differentiable without matching. The loss is

$$ \mathcal{L} = \lambda_{p}\,\| (\hat{x}-x)\odot b \|_1 + \lambda_{n} \sum_{i \in N_h} \mathrm{KL}\big(c_i(x)\,\|\,c_i(\hat{x})\big) + \lambda_{r} \sum_{(i,j)} \| r_{ij}(\hat{x}) - r_{ij}(x) \|_1 + \lambda_{g} \sum_{a \in A_h} s_a(\hat{x})\,[s_a(x)<\tau]. $$

The symbols are defined as follows. $b$ is a band around and inside the hole. $N_h$ is the set of clean-image objects whose boxes intersect the hole's neighborhood. $c_i(\cdot)$ is the detector's class distribution for region $i$. $r_{ij}(\cdot)$ is the relative offset between the refined boxes of objects $i$ and $j$. $A_h$ is the set of anchor locations inside the hole. $s_a(\cdot)$ is the detector's object score at anchor $a$. The final term is a ghost penalty that fires when the fill creates object evidence where the clean image had none, meaning a clean score below threshold $\tau$. We deliberately leave out a GAN term. The tradeoff paper's argument that perceptual realism can hurt machine readability suggests realism is not the target here.

The analysis focuses on stratification. Every evaluation is split by the fraction of an object's box covered by the patch: none, partial, and near-total. Objects that never overlap a patch measure collateral damage. The partial bin measures boundary repair. The near-total bin measures hallucination, and in that bin we report false-confident detections as well as recall.

Baselines

The naive reference points are the undefended detector and SAC-style zero-masking with the same localizer. Zero-masking shows how much any fill helps, and it is known to produce false TV and suitcase detections.

The direct predecessor is plain coherence transport with no learned correction. Jedi's pipeline built on it restores ImageNet accuracy to 64.34% from 39.26% under attack, as reported in An information-theoretic perspective of physical adversarial patches. Beating it on detection mAP in the partial-occlusion bin, at under twice its latency, is the minimum bar for the learned stage to be worth having.

The strongest accuracy competitor is diffusion inpainting as in DiffPAD. DiffPAD restores ConvNeXt on ImageNet from 3.9% to 82.2% under LaVAN, against 83.9% clean. Under white-box AdvP it holds 90.0%, against 88.3% for DIFFender.

The strongest detection system is PAD. It reports 83.23% mAP against naturalistic patches on Faster R-CNN and a 2.27% attack success rate on APRICOT.

For the traffic-scene setting, the competitor is RLID's learned inpainter, which restores YOLOv4 on TT100K from 0.18 to 0.51 mAP. Local gradient smoothing falls to 0.07 on the same benchmark.

Fairness depends on holding localization fixed. Each inpainter receives identical masks, drawn from ground-truth APRICOT-Mask masks, from SAC's segmenter, and from PAD's localizer. Differences then reflect filling alone. Learned baselines (RLID's inpainter and our network) get the same training images, pasted-patch distribution, mask-noise augmentation, and hyperparameter search budget. DiffPAD is run with its released step settings, and also at a reduced step count matched to our latency. Winning means higher mAP than DiffPAD and PAD's fill in the partial-occlusion bin at matched masks, with an equal or lower ghost-detection rate, at an order of magnitude less wall-clock time. If we match DiffPAD only at equal latency, we report it as a speed result, not an accuracy result.

Three ablations isolate the key ingredient. The first replaces the object-graph terms with a dense detector-backbone feature-matching loss of equal weight budget. The second removes the relation term. The third removes the ghost term. A further architectural ablation compares ungated, hard-gated (EMASC-style), and soft-ramp skip connections under noisy masks.

Why not just…?

Why not just use a detector-feature perceptual loss instead of an object graph? This is the most serious threat. SINCO shows that adding a downstream task loss to reconstruction preserves task-critical structure, and a dense loss on detector features is simpler than ours. Our argument is that dense feature matching weights every location equally, while detector failures concentrate at object boundaries crossing the hole and at spurious activations inside it. Our loss weights exactly those places. That argument is a hypothesis, and the first ablation tests it directly. If the dense loss matches us within noise in the partial bin and on ghost rate, we will adopt it and report the object-graph structure as unnecessary.

Why not just run diffusion inpainting, which already works? DiffPAD's 82.2% on ImageNet shows diffusion fills are strong. Both DiffPAD and DIFFender acknowledge that iterative denoising carries heavy latency, and generic priors fill holes with plausible but class-agnostic content. The latency-matched DiffPAD run shows whether diffusion's advantage survives the budget a real-time system can afford. The near-total-occlusion bin shows whether its realism comes with confident hallucinations.

Why not just correct the detector's outputs with scene context, as LOGICDEF does? LOGICDEF recovers 97% of clean accuracy on Visual Genome, but it corrects labels of objects that were detected. It cannot remove a ghost detection the black box or fill created, nor restore a box the patch truncated. It also depends on an upstream scene-graph extractor, which the patch itself can corrupt. Output correction is complementary rather than competing, so we also report LOGICDEF-style correction stacked on each inpainter's output.

Why not just adversarially train the detector? SAC reports that adversarial training of Faster R-CNN drops clean COCO mAP from 49.0% to 40.2%. LOGICDEF reports adversarial training recovering only about 42% of clean accuracy on Visual Genome. A plug-in preprocessor leaves the deployed detector untouched.

Proposed Experiments

The main digital experiment follows SAC's COCO protocol: 100×100 adaptive patches against Faster R-CNN, where SAC reports clean 49.0% mAP, 14.4% undefended, and 44.0% under its masking defense. We add PAD's naturalistic-patch setting. Physical evaluation uses APRICOT, with pixel masks from APRICOT-Mask (APRICOT). There SAC reduces targeted success from 7.97% to 2.17% and PAD reports 2.27%. We also test TT100K with YOLOv4 for small-object driving scenes. For classification transfer we test ImageNet under LaVAN, which is mainly a check that we do not lose much to DiffPAD there. Metrics are mAP and attack success rate overall and per occlusion bin, ghost-detection rate inside holes, clean-image mAP when masks are false positives on benign textured regions, PSNR on the hole band, and milliseconds per image on a single GPU and on an embedded board.

Three checks decide whether the idea holds.

The first is cross-detector transfer. The loss uses one frozen Faster R-CNN, so we evaluate the trained inpainter in front of RetinaNet, SSD, and YOLO without retraining. If gains disappear on unseen detectors, we have overfit the inpainter to one detector's quirks. That would be a real limitation and would change the claim from "plug-in defense" to "detector-specific fill".

The second is an adaptive attack. The learned residual is differentiable, so we attack the full pipeline end to end. The localizer is approximated with BPDA, a backward-pass differentiable approximation that substitutes a smooth surrogate for gradients through non-differentiable steps. The attacker optimizes the patch to survive localization and to steer the residual into creating a target object. We report robust mAP alongside DiffPAD's white-box figure from the file as a reference point, keeping in mind it is a classification number.

The third is the near-total-occlusion bin. A good result there is low confident false positives, not high recall, consistent with the information-loss argument. If our inpainter "recovers" fully hidden objects at rates above what an occlusion-matched clean image allows, the loss is teaching it to guess. We would then strengthen the ghost term rather than celebrate the result.