Gradients That See the Grid: Cheap Adversarial Training for Cylindrical LiDAR Networks

Self-driving cars commonly label every LiDAR point (road, car, pedestrian, pole) with networks that first sort the points into cylindrical cells. Cells are cut by distance from the sensor, by angle around it, and by height, so they grow with range the way the laser beams spread. Nobody has measured how these networks hold up against deliberately crafted point perturbations. Defending them with adversarial training is very expensive. Adversarial training means training on inputs that an inner optimizer has nudged to maximize the loss. With $K$ attack steps it costs roughly $K+1$ times ordinary training, and a single forward pass of Cylinder3D already takes 170 ms. The standard way out is single-step adversarial training, as in Fast is better than free and the Fast Newton Method of Single-Step Adversarial Training for Semantic Segmentation. These methods compute one gradient of the loss with respect to the input.

We argue this gradient is incomplete in a cylindrical network. A point's coordinates enter the network in two ways. They are features fed to a small per-point network, which is differentiable. They also decide which cell the point falls in, a floor operation whose gradient is zero. The input gradient therefore sees only perturbations that keep each point inside its cell. It is blind to perturbations that push points across cell boundaries. Because angular cells are only a few centimeters wide near the sensor and meters wide far away, that blind spot depends on range. Our proposal makes the attack used during training see the grid. It does so by spreading each point softly over neighboring cylindrical cells during attack generation only, and by setting the step size separately for each range ring. The goal is robustness at close to the cost of one extra gradient pass.

Core Claim

We make two claims, and both can be falsified. The first is that single-step or multi-step adversarial training through the hard cylindrical binning produces a recognizable kind of false robustness. The trained model resists gradient attacks but stays vulnerable to attacks that flip points across cell boundaries, and the vulnerability is concentrated near the sensor and on thin objects. The second is that a single-step attack computed through a soft, range-aware cylindrical assignment closes most of this gap. It should match multi-step adversarial training through the same soft assignment while costing about two times clean training rather than $K+1$ times.

The soft assignment is ordinary trilinear splatting in cylindrical coordinates, with the angle axis wrapping around at 360°. A point with continuous cylindrical coordinates $u=(\rho,\theta,z)$ contributes to each neighboring cell $c$ with weight

$$w_c(u)=\prod_{d\in\{\rho,\theta,z\}}\max\!\left(0,\,1-\frac{|u_d-c_d|}{\Delta_d}\right),$$

where $c_d$ is the cell center along axis $d$ and $\Delta_d$ is the cell size along that axis. These weights are differentiable in $u$, so the attack gradient includes the effect of moving a point into the next cell. The model's forward pass at test time is unchanged and still uses hard cells. The soft version is a training-time device only.

The second ingredient is the step size. A Cartesian perturbation of size $\epsilon$ moves a point across an angular cell only when $\epsilon$ is comparable to the cell's arc length $\rho\,\Delta\theta$, and that arc length grows linearly with range $\rho$. The Fast Newton Method sets one step size from a classification margin averaged over all pixels. We instead compute that margin separately in radial rings and solve for one step size per ring. A single scan-wide average would be dominated by the dense near-range points.

Relation to Prior Work

Two lines of work motivate this proposal. Cylinder3D and PolarNet show that cells matched to the sensor raise non-empty cell occupancy from 61% to 89%. They lift SemanticKITTI mIoU to 67.8% (Cylinder3D) and 54.3% (PolarNet), against 49.9% for range-image projection in SemanticKITTI. Neither paper studies adversarial inputs. Physically Realizable Adversarial Examples for LiDAR Object Detection found that Cartesian occupancy-voxel detectors are more fragile than point-based ones. It also found that a costly mesh-augmentation defense still left a 5–20% attack success rate. Cylindrical grids were never tested. Our first experiment fills that gap before any defense is built.

Three papers supply methods. We reuse random initialization from Fast is better than free, which is what prevents catastrophic overfitting (the collapse in which single-step training looks robust but has near-zero robustness to multi-step attacks). We adapt the margin-based step-size rule of the Fast Newton Method from 2D pixels to radial rings. We borrow the wrap-around angular treatment from PolarNet's ring convolution for the soft splatting. All three earlier methods assume the input is a continuous tensor that the network reads directly. Our contribution starts from the observation that LiDAR grid networks violate that assumption.

Point-cloud defenses constrain the design. Minimal Adversarial Examples for Deep Learning on 3D Point Clouds changes about 3.5% of points to reach 89.38% attack success on ModelNet40 while evading outlier removal. Any filtering defense therefore has to be tested against on-surface attacks. IF-Defense restores points through implicit surfaces at 1.0–1.6 s per object, far beyond a 100 ms frame budget. PointCAT adds a learned noise generator and contrastive losses but works on object classification with PointNet. We do not compete with these as point-cloud defenses. They tell us that a scene-scale defense must live inside training and leave inference cost unchanged.

Research Program

The threat model perturbs the coordinates of existing points within a per-point $\ell_\infty$ ball of a few centimeters. We sweep $\epsilon$ from 2 to 10 cm, which is on the order of sensor noise. We report attacks on all points and attacks restricted to points of one target object. Point addition is a secondary threat model, evaluated but not trained against.

The work has three stages. First, we build an evaluation attack suite that does not share the defense's surrogate. It contains multi-step attacks through hard binning, through the soft splat, and through a straight-through estimator of the floor. It also contains a gradient-free boundary-flip search. That search takes points within $\epsilon$ of a cell boundary, moves them across, and greedily keeps flips that raise the loss. We report the worst case per scan. Second, we measure undefended and naively defended cylindrical, polar, and Cartesian-voxel networks under this suite, broken down by range ring and by class. Third, we train the proposed single-step soft-splat, ring-adaptive defense. We report robustness together with wall-clock GPU-hours relative to clean training.

Baselines

The naive reference point is undefended Cylinder3D trained with its usual augmentation. Its reported clean numbers are 67.8% mIoU on the SemanticKITTI leaderboard and 76.1% mIoU on nuScenes (Cylinder3D). We found no reported result on Lacuna for any cylindrical or polar LiDAR segmentation network under adversarial point perturbation, so the robust numbers for this baseline are something we must measure. A second trivial reference adds random coordinate jitter of the same magnitude $\epsilon$ during training. This separates real adversarial training from noise augmentation.

The strongest cheap competitor is single-step training with random initialization in the style of Fast is better than free, with gradients flowing through the hard binning via the per-point feature path. We also run its step-size-controlled variant from the Fast Newton Method (Single-Step Adversarial Training for Semantic Segmentation). These are the obvious things a practitioner would try, and the original papers report that they match multi-step training in 2D. For CIFAR-10, the fast method reaches 45% robust accuracy in 6 minutes versus 80 hours for multi-step training. The Fast Newton paper gives no numeric figures for Cityscapes, so no reported result was found on Lacuna for its segmentation mIoU.

The strongest expensive competitor is multi-step training through hard binning with $K=7$. Because a full schedule may be unaffordable, we run it on the full data with a shortened schedule and, separately, on a subset for the full schedule. We also run multi-step training through the soft splat as the upper reference for our method.

All methods share the Cylinder3D architecture, data splits, epochs, mixed precision, and a fixed tuning budget of the same number of trial runs for $\epsilon$ and step size. Beating the field means two things. Our method must reach worst-case robust mIoU within a small margin of multi-step soft-splat training, and clearly above both hard-binning single-step methods. It must also cost no more than about 2.5 times clean training, with clean mIoU loss no larger than the hard-binning methods.

Two ablations isolate the ingredients. One uses soft splatting with a single scan-wide step size. The other uses ring-adaptive steps with hard binning. We repeat the main comparison on PolarNet to test whether the effect depends on 3D cells or holds for polar bird's-eye-view cells too.

Why not just…?

Why not just run multi-step adversarial training offline, since compute is plentiful? At 170 ms per Cylinder3D forward pass and with 43,000 SemanticKITTI scans, each attack step adds a forward and a backward pass per scan per epoch. A $K=7$ schedule is therefore close to an order of magnitude more expensive than clean training. More importantly, our first claim predicts that multi-step attacks through hard binning inherit the same blind spot as single-step ones. If so, extra compute buys no defense against boundary flips. The hard-binning multi-step baseline tests this directly. If it turns out robust under the boundary-flip search, the blind-spot hypothesis is wrong and only the cost argument remains.

Why not just use a straight-through estimator for the floor instead of soft splatting? This is a real threat, because the straight-through estimator is cheaper and simpler. Its gradient treats the cell index as continuous but ignores where the boundary actually sits. Our prediction is that it gives poorly aimed boundary-crossing directions near the sensor, where cells are smallest. We include it both as an evaluation attack and as a training variant. If it matches soft splatting, we will recommend it and report the soft splat as unnecessary.

Why not just add outlier removal or random jitter at inference? Minimal on-surface attacks evade outlier removal (Minimal Adversarial Examples). The randomization defense of Mitigating Adversarial Effects Through Randomization is known to fail against attackers who average over the random transformations. We evaluate jitter-at-inference under exactly such an averaging attack. We expect it to fall to near-undefended robustness while costing clean accuracy on sparse distant points.

Why not just freeze the backbone and train a small robustness add-on, as with ROB tokens? Robustness Tokens recovers ADE20K mIoU from 4.7 to 23.3 in 200 steps. However, it is built on transformer token sequences that sparse 3D convolutions lack, and part of its protection depends on keeping the tokens secret. It addresses training cost but not the boundary-flip blind spot, so we treat it as out of scope rather than as a competitor.

Why not just switch to a 2D range-image network where fast adversarial training is solved? Range-image projection loses 8–17 mIoU points against Cylinder3D on SemanticKITTI (Cylinder3D), which is too high a price for robustness. A range image also has its own discrete pixel assignment, so it would likely share the blind spot.

Proposed Experiments and Decisive Checks

Experiments use SemanticKITTI single-scan segmentation (validation sequence 08 for attacks, since test labels are hidden) and nuScenes LiDAR segmentation. Metrics are clean mIoU, worst-case robust mIoU over the attack suite at each $\epsilon$, robust IoU for thin classes (pole, traffic sign, motorcyclist), robust mIoU per 10 m range ring, and training GPU-hours relative to clean training. We run three seeds per setting and record how often catastrophic overfitting occurs across seeds and $\epsilon$ values.

Four checks decide the outcome. The first is the blind-spot test: for models trained through hard binning, does robust mIoU under the boundary-flip search fall well below robust mIoU under gradient attacks, with the drop concentrated in the near-range rings? If not, the core claim fails. The second is the main result: does single-step soft-splat, ring-adaptive training close that drop to within a few points of multi-step soft-splat training at roughly one quarter of its cost? The third is attribution: the two ablations and the straight-through variant show which ingredient carries the gain. The fourth is representation: under matched training, do cylindrical cells end up more or less robust than Cartesian cells? This answers the open question of whether large distant cells dilute a perturbation or amplify it.