Robots that pick parts out of bins estimate an object's six-degree-of-freedom pose (its 3D position and orientation) from a camera, and the best methods for cluttered, partly hidden objects work by voting. Every visible point predicts where a handful of the object's reference points ("keypoints") should be, and a consensus step pools those predictions. These systems handle physical occlusion well, but nobody has checked how they respond to adversarial inputs. The usual fix for adversarial inputs, adversarial training, re-optimizes a worst-case perturbation at every training step. That multiplies cost several times over even with the fastest single-step shortcuts, and those shortcuts were only ever validated on image classification.
This note argues that for one realistic threat, a sticker-like adversarial patch confined to a region of the image, the expensive training loop can be skipped entirely. The argument rests on a property called local support: each vote should depend only on a small neighborhood of pixels and points. If that holds, a patch can only corrupt the votes whose neighborhoods it touches, which is exactly what an occluder does. The patch attack then reduces to an occlusion problem. Occlusion can be trained against with ordinary random erasing at roughly the cost of standard training, and the damage the patch can do can be bounded by counting pixels rather than by running an attack. The catch is that current voting networks do not have local support: their backbones pool global context. So the proposal is to build a voting network that has it, measure what that costs in clean accuracy, and certify it.
Core Claim
Suppose each point $p$'s vote is computed only from pixels and depth points within radius $r$ of $p$, and the attacker may modify any pixels inside a region $R$. Then the votes that can change are exactly those in the set $V(R)$ of points whose radius-$r$ neighborhood intersects $R$. For all other points, the vote under attack is identical to the clean vote.
Now let the consensus step be a robust aggregator with breakdown fraction one half, such as the geometric median of per-point keypoint predictions. Whenever the corrupted fraction $\alpha(R) = |V(R)|/|V|$ stays below one half, each keypoint estimate stays inside a ball. That ball's radius depends only on $\alpha(R)$ and the spread of the uncorrupted votes, following standard breakdown bounds for the geometric median. A least-squares rigid fit then turns bounded keypoint errors into bounded pose errors.
The certificate therefore needs two things. First, the worst-case footprint $\max_R \alpha(R)$ over all patch placements, which is pure geometry. Second, the clean votes, which come from a single forward pass. No gradient-based attack is needed at training time or at certification time.
The claim is new in a specific way. Prior work treats local support as protection against occlusion and treats adversarial robustness as an optimization problem. Here local support is the mechanism that converts a bounded-region adversary into an occluder, which turns robustness into a counting argument.
Why current voting networks fail this test
The motivating system is the 3D Point-to-Keypoint Voting Network. It fuses per-point RGB features from a PSPNet-style image network with depth features from a PointNet-style network, votes for keypoints chosen by farthest point sampling on the CAD model, and aggregates the votes by RANSAC. It reports 98.7% ADD(-S) on LINEMOD and 52.6% on OCCLUSION LINEMOD. ADD(-S) is the fraction of frames whose average model-point distance falls under 10% of the object diameter.
Its occlusion robustness comes from voting, but neither backbone has local support. PSPNet's pyramid pooling and PointNet's global max-pooled feature both let any pixel influence every vote. A patch on the table next to the object can therefore shift all votes coherently. Towards Adversarial Robustness via Feature Matching documents the classification analogue of this, an "attention shift" toward background pixels, and counters it with a regularizer that still requires 7-step PGD.
Coherent shifts are exactly what consensus cannot reject. RANSAC Revisited shows that classical RANSAC degrades under coordinated rather than random corruption. Coordinated corruption is what a targeted attack produces: Adversarial Attacks on Monocular Pose Estimation reports an 18.46× increase in rotation error on KITTI from an "Invert Yaw" attack on globally regressing networks, and leaves geometry-aware defenses explicitly open.
The design principle is old. A Representation for Image Curves argued that representations built from critical points leave visible segments unaffected by occlusion elsewhere. Morphological Shape Context restricted shape context to meaningful boundaries so that peripheral clutter cannot corrupt a descriptor. The line-based detector of Cvetković et al. recovers pose from a single matched line under 90% occlusion. These works supply the principle. We apply it to learned per-point features and to an adversarial rather than a physical occluder.
Research Program
The local-support voting network. We build a network whose per-point outputs have a provably bounded receptive field. The RGB branch is a convolutional network with no global pooling, no attention, and no normalization computed over the whole image. We use frozen batch-norm or group statistics computed per patch, and record the exact receptive field. The depth branch is a PointNet++-style stack of radius-limited neighborhood aggregations with the global feature removed. We fuse the two branches per point, as in the original network. For a clean certificate, the voting head predicts 3D offsets to each keypoint rather than unit directions, so that each point contributes one candidate keypoint position. The aggregator is the geometric median, and pose comes from the same least-squares fit.
Training. The network is trained only with occlusion-style augmentation: random erasing of image rectangles and point dropping, as in the original network's spatial dimension sampling. Its cost is benchmarked against standard training in GPU-hours.
Certification and attack. For each test frame and patch size $s$, we compute $\max_R \alpha(R)$ by sliding the patch footprint over the image. This is a counting operation over pixel positions. We then evaluate the keypoint and pose bound and report certified ADD(-S): the fraction of frames whose worst-case pose is guaranteed to fall within the 10%-diameter threshold. Separately, we run strong adaptive patch attacks, many-step PGD on patch contents over sampled locations, to confirm that empirical robustness sits above the certificate.
Scope limits, stated in advance. Local support does nothing against a full-image $L_\infty$ perturbation, because such an attack touches every neighborhood and can still coordinate the votes. We will show this as a negative control rather than claim it. We also flag the main risk. Removing global context may cost clean accuracy on OCCLUSION LINEMOD, where context helps disambiguate short, uninformative visible fragments. That is the "primitive ambiguity" problem this literature has long noted. Measuring this robustness tax is a central result, not a footnote.
Baselines
Undefended reference. The naive reference point is the original point-to-keypoint voting network, retrained by us on LINEMOD and OCCLUSION LINEMOD with its published recipe. Its reported clean numbers are 98.7% (LINEMOD) and 52.6% (OCCLUSION LINEMOD) ADD(-S). We first reproduce these so that the local-support variant's clean-accuracy tax is measured against a verified figure. For patch robustness, the proposal succeeds only if it beats this model's attacked ADD(-S) by a wide margin while losing a bounded amount of clean ADD(-S). We treat a loss of under five points on OCCLUSION LINEMOD as acceptable. No reported adversarial result for this network exists on Lacuna.
Strongest existing competitor: patch adversarial training. This is multi-step adversarial training, in the min-max formulation described in the research file, applied to the original network with patches as the inner adversary. It directly targets the threat and is the gold standard defense. We also run the cheaper single-step variant from Fast is better than free, which uses random initialization inside the perturbation set. That paper reports 45% PGD robust accuracy on CIFAR-10 in 6 minutes versus 80 hours for PGD training. Free adversarial training reports about 46% in about 13 hours on the same benchmark. No reported result exists on Lacuna for either method on pose estimation.
Fairness rules for these comparisons are as follows. All methods get the same data, the same backbone capacity (parameter count within 10%), and the same hyperparameter search budget. Training GPU-hours are reported for every method. The proposal "beats" these baselines if it matches their empirical ADD(-S) under our strongest adaptive patch attack at a fraction of their training cost, while also providing a certificate they cannot.
Ablations isolating the key ingredient. Two crossed ablations separate locality from robust aggregation. The first uses the original global-context backbone with the geometric-median aggregator and random erasing. The second uses the bounded-receptive-field backbone with ordinary RANSAC. A third ablation, the bounded backbone without occlusion augmentation, tests whether occlusion training is what keeps the clean votes' spread small enough for the bound to be useful.
Why not just…?
Why not just train with random erasing and keep the original network? This is the most serious threat, because random erasing is exactly our training recipe. The difference is the backbone. With global pooling, a patch outside the erased-region distribution can still move every vote, and nothing bounds its reach. We will measure this directly as a "vote footprint" curve: the change in each point's vote as a function of its distance from the patch, under attack, for both backbones. If the global-context network with random erasing and median voting matches our empirical robustness under adaptive attack, locality is not doing the work. In that case we would report the result and fall back to the certificate as the sole contribution.
Why not just apply fast single-step adversarial training? It may work empirically, and we run it. Two points count against it. It was validated only on cross-entropy classification. Its known failure mode, collapse to robustness against the single step only, has not been characterized for continuous vote-regression losses. It also costs a multiple of standard training and yields no guarantee. If it matches us empirically at close to our cost, the certificate remains the distinguishing contribution.
Why not rely on RANSAC to discard corrupted votes? RANSAC Revisited shows that consensus breaks under coordinated corruption. The "bounded backbone plus RANSAC" ablation tests how much of our effect comes from the aggregator versus the bounded footprint.
Why not use input randomization or robustness tokens? Random resizing and padding from Xie et al. alter camera geometry, which corrupts pose regression directly. That paper itself notes vulnerability to attacks that average gradients over the random transformations. Robustness Tokens require a frozen vision transformer, and the voting network is not one. Adapting them would change the architecture being compared, so we leave them out.
Decisive Experiments
All experiments run on LINEMOD and OCCLUSION LINEMOD, with RGB patches of side 16 to 64 pixels and matched depth-region perturbations. We report clean ADD(-S), attacked ADD(-S), certified ADD(-S), rotation and translation error multipliers (mirroring the monocular attack paper), and training GPU-hours.
Four outcomes decide the proposal. First, the bounded network's certified ADD(-S) should be nonzero and grow as the patch shrinks. Second, its empirical attacked ADD(-S) should rival multi-step patch adversarial training at roughly standard-training cost. Third, the vote-footprint curve should drop to zero beyond radius $r$ for the bounded network but not for the global-context one. Fourth, the $L_\infty$ control should show no advantage, which confirms that the mechanism is locality rather than generic regularization.
If the clean tax on OCCLUSION LINEMOD exceeds the acceptable margin, the finding becomes a measured trade-off curve: receptive-field radius against certified robustness and clean accuracy. That curve is itself the first quantitative answer to how much context a voting pose estimator can afford to give up.
References
- 3D Point-to-Keypoint Voting Network for 6D Pose Estimation
- Towards Adversarial Robustness via Feature Matching
- RANSAC Revisited: An Improved Algorithm for Robust Subspace Recovery under Adversarial and Noisy Corruptions
- Adversarial Attacks on Monocular Pose Estimation
- A Representation for Image Curves
- Morphological Shape Context: Semi-locality and Robust Matching in Shape Recognition
- DETECTION OF TEXTURE-LESS OBJECTS BY LINE-BASED APPROACH
- Fast is better than free: Revisiting adversarial training
- Adversarial training for free!
- Mitigating Adversarial Effects Through Randomization
- Robustness Tokens: Towards Adversarial Robustness of Transformers