Self-driving cars increasingly perceive the world as a 3D grid of small cubes, called voxels, each labeled free or occupied by a class such as car, road, or building. This task, 3D occupancy prediction, replaced flat bird's-eye-view maps because those maps discard height, which matters for overpasses and hanging obstacles. Every published occupancy result, however, comes from train/test splits of nuScenes and Waymo that were cut by time, not by place. For flat maps built by the same camera systems on the same datasets, place-based splitting has already been shown to be decisive. More than 80% of nuScenes validation samples lie within 5 meters of a training sample, and MapTRv2 falls from 72.7 to 26.2 mean average precision once training and test locations are separated (Localization is All You Evaluate). The models were largely recognizing where they were and recalling the map. Nobody has checked whether the same holds for 3D occupancy.
This note proposes that check, together with two things that make it more than a rerun. First, occupancy labels are built by fusing LiDAR from many frames along a drive. Those labels therefore carry geometry far from the car's own position. This opens a leakage path that does not exist for flat maps and that a buffer sized to the network's receptive field will not close. Second, we break the error down by voxel. Static versus moving classes, and visible versus occluded voxels, show where memorization lives. The question is whether keeping the 3D structure makes a network infer geometry from what the cameras see, or whether it simply gives the network more room to memorize the scene.
Core Claim
We claim that a large share of reported occupancy accuracy on nuScenes is recall of location. That share should be concentrated in static classes (drivable surface, sidewalk, terrain, man-made structures) and in voxels the cameras cannot see. Moving classes such as cars and pedestrians cannot be memorized by location, so their accuracy should hold up. A fair geographic split for occupancy must also exclude label leakage: supervision voxels in training frames that physically lie inside the test region. It is not enough to separate camera poses. The second, comparative claim is open in sign. It asks whether lifting image features into a true 3D volume, as SurroundOcc does, loses less accuracy under geographic shift than occupancy decoded from a flat bird's-eye-view feature, as BEVFormer does. Either answer changes how the field should read occupancy leaderboards.
Research Program
The first component is a set of place-disjoint splits for Occ3D-nuScenes (Occ3D) and for the SurroundOcc labels (SurroundOcc). We reuse the two protocols of Lilja et al. without change. The first is Near Extrapolation: disjoint zones within one city, with a 60-meter buffer. The second is Far Extrapolation: train in one city (for example Boston) and test in another (Singapore). Their paper supplies the method and motivates the whole study. What it does not do is handle volumetric labels.
We then add the occupancy-specific fix. Any training-frame label voxel whose world coordinates fall inside a test zone or its buffer is masked out of the loss. The alternative would be to widen the buffer to the full horizontal extent of the label grid. We report both options, because masking keeps far more training data. The idea of sizing buffers to what the model can "reach" comes from the leakage-safe protocol for subglacial bed mapping (Learning Subglacial Bed Topography). That work sized buffers to the convolutional receptive field on a 2D raster. We extend the reach to include label aggregation, which is where volumetric data actually leaks.
The second component is a continuous leakage curve. It complements the discrete splits and answers the cheapest objection. Each standard-split validation frame is binned by the distance from its ego pose to the nearest training pose. We then plot mean intersection-over-union (mIoU, the standard occupancy score) against that distance. Hard splits give a few points on the curve; the curve shows the dose-response.
The third component is the voxel-level breakdown. Occ3D provides camera visibility masks, so we compute accuracy separately for four cells: static classes in visible voxels, static in occluded, dynamic in visible, and dynamic in occluded. We define the memorization gap of a stratum as its mIoU on the standard split minus its mIoU on the geographic split. Our hypothesis predicts a clear order of gaps: largest for static-occluded voxels, smallest for dynamic-visible ones. Lilja et al. observed qualitatively that flat-map models stop "seeing through" large obstacles on disjoint splits. This breakdown turns that observation into a number in 3D.
The fourth component compares three architectures under matched conditions:
- SurroundOcc. Queries arranged as a 3D grid pull features from the images, followed by a 3D U-Net.
- CTF-Occ from Occ3D. It attends in 3D only to selected voxels, coarse to fine.
- A BEVFormer occupancy head. Height is collapsed into a flat plane and recovered at decoding.
This contrast isolates whether a native volumetric representation, rather than more parameters, changes the memorization gap.
The logic plausibly carries over to medical volumes. There, overlapping 3D patches from one scan play the role of nearby poses, and the models in MA-SAM and the 3D Mamba analysis were evaluated with patient-level random splits only. We leave that for follow-up rather than diluting the driving study.
Baselines
The naive reference point is each model trained and tested on the standard nuScenes split, as published. For Occ3D-nuScenes, CTF-Occ reports 28.53 mIoU and BEVFormer 26.88 mIoU (Occ3D). On its own labels, SurroundOcc reports 20.30 semantic scene completion mIoU and 31.49 geometry IoU (SurroundOcc). We found no reported result on Lacuna for any occupancy model on a geographically disjoint split. Every geographic number in this study will therefore be new, and we will reproduce the standard-split numbers first, to within a point, before trusting our pipeline.
The strongest existing volumetric competitor is CTF-Occ, the best standard-split score in the file. The strongest volumetric-lifting design is SurroundOcc. All models are retrained from the same image backbone initialization on identical frame lists for each split. They get the same number of training iterations and the same augmentation, and hyperparameters are tuned only on a validation zone carved from the training region, never on the test zone.
For context from flat maps, we cite the gap sizes Lilja et al. measured: MapTRv2 goes from 72.7 to 26.2 on Near Extrapolation, and from 65.3 to 55.2 on Argoverse 2. UIGenMap reaches 39.8 mAP on the nuScenes region split (UIGenMap). These tell us whether 3D occupancy degrades more or less than vector maps do.
The ablation that isolates our key ingredient is the geographic split run twice: once with pose-only buffering and once with label-voxel masking. A drop between the two is direct evidence of label leakage. A "beating" result for the volumetric hypothesis means that SurroundOcc's static-occluded memorization gap is smaller than BEVFormer's by more than the spread across three training seeds.
Why not just…?
Why not just rerun SurroundOcc on the Lilja splits and report one number? That is the obvious first experiment, and it is contained here. Alone, though, it would miss two things. Pose-based buffers can leave label leakage in place, so a small drop could be a false reassurance. The masking ablation is designed to catch exactly that. A single mIoU also cannot say whether accuracy fell because of memorization or because of ordinary appearance shift, such as Singapore's left-hand traffic or different vegetation. The static-versus-dynamic breakdown separates the two, because dynamic objects cannot be memorized by location.
Why not just filter the standard validation set by distance to training poses? This is cheap and genuinely useful, which is why the leakage curve is a core output. It still inherits label leakage from training frames near the boundary. It also cannot produce a far-extrapolation regime, because nuScenes validation frames are almost all close to training frames. If the curve and the masked split agree, the cheap method is vindicated and we will say so.
Why not just add pose noise or heavier augmentation so the model cannot localize? Memorized scene layout can be recalled from appearance alone, and perturbing poses does not remove the appearance cue. Lilja et al. found that the gap persists in models trained with standard augmentation. We will nonetheless include one run with strong photometric and extrinsic augmentation. If it closes the static-occluded gap, that is an important and cheap remedy, and it would weaken the architectural claim.
Proposed Experiments
The decisive check comes first: CTF-Occ on Occ3D-nuScenes Near Extrapolation with label masking. If its static-occluded mIoU falls by a large fraction while dynamic-visible mIoU stays within a few points, the memorization claim holds. If both fall similarly, the drop is generic domain shift, and the memorization claim fails.
The second check is the masking ablation. Any extra drop from masking training labels that lie inside the test region quantifies leakage specific to 3D labels.
The third check compares SurroundOcc, CTF-Occ, and BEVFormer gaps across three seeds, on Near and Far Extrapolation. Far Extrapolation also runs on Occ3D-Waymo where location metadata permits; CTF-Occ's standard-split reference there is 18.73 mIoU.
The fourth check is the distance-binned leakage curve on the standard split, compared against the hard-split numbers.
The main cost is retraining three heavy occupancy models on several splits. We keep the backbone fixed across models to contain it.
References
- Localization is All You Evaluate: Data Leakage in Online Mapping Datasets and How to Fix it
- Occ3D: A Large-Scale 3D Occupancy Prediction Benchmark for Autonomous Driving
- SurroundOcc: Multi-Camera 3D Occupancy Prediction for Autonomous Driving
- Learning Subglacial Bed Topography from Sparse Radar with Physics-Guided Residuals
- MA-SAM: Modality-agnostic SAM adaptation for 3D medical image segmentation
- A Comprehensive Analysis of Mamba for 3D Volumetric Medical Image Segmentation
- Uncertainty-Instructed Structure Injection for Generalizable HD Map Construction