A self-driving car's camera network spends most of its compute re-describing a world that has barely changed. Between two camera frames 80 milliseconds apart, the road, buildings and parked cars are physically identical. They still occupy different pixels, though, because the car moved. Existing "reuse what didn't change" methods for vision transformers compare image patches at the same pixel location over time. On a moving platform this comparison is defeated: the whole image shifts, every patch looks new, and nothing gets skipped. We propose to index the reuse cache by where a patch is in the 3D world rather than where it sits in the image. LiDAR depth tells us exactly where each previously computed patch feature should land in the current frame. The network then recomputes only patches whose content is plausibly new: surfaces never seen before, objects that moved, and, as a hypothesis we test rather than assume, surfaces now seen from a substantially different angle.
The last condition is where this work parts ways with the closest prior method. Seeing Once is Enough? Online Geometry-Aware Token Pruning for 3D Question Answering drops any image token whose 3D voxel was already observed. Long-standing results on appearance say this should fail in a specific, measurable way. Changing viewpoint or lighting can move an image more than changing the object's identity does (Face Recognition: the Problem of Compensating for Changes in Illumination Direction; Exploiting View-Specific Appearance Similarities Across Classes for Zero-Shot Pose Prediction). The goal is a camera backbone that costs a fraction of its full price inside a LiDAR–camera detector, with detection quality that stays put. Along the way we measure when, in real driving data, a re-observed surface actually carries new information.
Core Claim
For multi-sensor perception on a moving vehicle, temporal reuse of camera features has two requirements. First, it must be motion-compensated in 3D using the other sensor's depth. Second, its refresh rule must depend on how the observation changed geometrically, not only on whether the location was seen before. Concretely, each cached token stores the 3D surface point it came from, a local surface normal estimated from nearby LiDAR returns, and the viewing direction at the time it was computed.
At a new frame, cached tokens are projected into the image using LiDAR depth and ego-motion. For each token $i$ we form four quantities. The first is the residual $r_i$, the difference between the current cheap patch embedding and the warped cached one. The second is the change in viewing angle $\Delta\theta_i = \arccos(\mathbf{v}_t \cdot \mathbf{v}_{\text{ref}})$, where $\mathbf{v}_t$ and $\mathbf{v}_{\text{ref}}$ are unit vectors from the camera to the surface point now and when the token was cached. The third is a disocclusion flag $o_i$ marking image regions that no cached surface covers. The fourth is a dynamic-object flag $d_i$ from the previous frame's detections. A small learned gate turns these into a refresh score:
$$ s_i = g_\phi(r_i, \Delta\theta_i, o_i, d_i), $$
and the top-$k$ tokens by $s_i$ are recomputed under a fixed per-frame budget $k$. Here $g_\phi$ is a two-layer network with parameters $\phi$. The remaining tokens reuse their warped cached values through the delta-attention mechanism of Eventful Transformers.
What would be new has two parts. One is the surface-indexed, LiDAR-warped cache inside a fusion detector. The other is an empirical test of whether viewing-angle change predicts deep-feature change beyond what the warped residual already reveals. If it does not, we say so, and the result is a clean 3D-compensated version of event-driven reuse.
Research Program
The three input directions play different roles here. Temporal redundancy supplies the goal and the reuse machinery: token gating and incremental attention, taken from Eventful Transformers. That paper reports a 3.8× compute reduction (467 to 122 GFlops) for a 1.68% mAP drop on ImageNet VID, a benchmark with little calibrated ego-motion. Cross-modal fusion supplies the mechanism that makes reuse possible on a moving car: LiDAR points projected into the image give per-patch metric depth. This is the projection that PointFusion-style methods (MVX-Net) use to attach image features to points, run here in the opposite direction to move features between frames. Viewpoint-dependent appearance supplies the refresh criterion and the scientific question.
The host system is BEVFusion. It has decoupled camera and LiDAR streams that merge in a bird's-eye-view grid (an overhead map of the scene), and its paper reports 69.2% mAP on nuScenes. We use its camera stream because that paper and BEVFormer both identify the image backbone as the computational bottleneck. Token gating needs global-attention tokens, so we first retrain BEVFusion with a plain vision-transformer image backbone. That full-compute model is our reference; the original Swin-backbone figure is context only.
The design must handle three practical problems. LiDAR is sparse at range, so patches with no projected returns use the rotation-only warp (exact for distant points). We then measure separately how much error that approximation introduces. Cached errors also accumulate, a limitation the research file flags for event-driven buffers. We therefore track each token's age and include a periodic full-refresh schedule as a tunable knob, reporting the accuracy cost of removing it. Finally, camera and LiDAR are not perfectly synchronized, and AsyncBEV shows that sensor lag causes misalignment that hurts moving objects. We therefore stress-test warping with artificially delayed LiDAR depth.
The gate $g_\phi$ is trained against the full-compute model with the backbone frozen. The training target is the per-token change in final-layer features after warping, so the gate learns to predict which tokens would actually differ. This regression also serves as our diagnostic. Fitting it with and without $\Delta\theta$ directly measures how much feature change viewing-angle change explains beyond the residual.
Baselines
The naive reference point is the full-compute vision-transformer BEVFusion run on every camera frame. It defines the accuracy ceiling and the compute we are trying to cut. Every other method is compared at matched camera-backbone GFLOPs per frame, on the same nuScenes splits, starting from the same trained weights. Each method's thresholds or budgets are tuned on the same validation scenes with the same number of tuning runs.
The strongest existing reuse method is Eventful Transformers applied unchanged to the camera backbone, with same-pixel reference tokens and the top-$r$ gate it describes. It is the strongest competitor because it is the only cited method that cuts per-frame transformer compute without retraining, and it reports the 3.8× / 1.68% figure above on ImageNet VID. There is no reported result found on Lacuna for Eventful Transformers on nuScenes or on any ego-moving multi-camera benchmark. We run it ourselves at five budgets.
The second competitor is the "seeing once is enough" rule ported from Seeing Once is Enough?: any token whose 3D voxel was already observed is reused, regardless of angle. On ScanQA with Qwen2.5-VL-7B that paper reports Exact Match rising from 24.1 to 25.1 with up to 50% token reduction. In our system this is exactly our cache with $\Delta\theta$ and $r$ removed from the gate, so it doubles as an ablation.
The third competitor is camera-rate reduction with temporal warping. The camera backbone runs every $n$-th frame, its bird's-eye-view features are carried forward by ego-motion as in BEVFormer's temporal self-attention, and LiDAR runs every sweep.
The key ablation is the warped-residual-only gate: 3D-warped cache with $g_\phi$ seeing only $r_i$. Beating it isolates the contribution of geometric refresh signals.
Success means one of two outcomes against both Eventful Transformers and warped-residual-only. Either our method keeps nuScenes Detection Score (NDS, nuScenes' combined detection metric) within 0.5 points of full compute at a lower budget than either competitor, or it achieves higher NDS and lower velocity error at equal budget. Against frame-rate reduction, success means a clear gain on dynamic-object and newly revealed-region recall at matched compute.
Why not just…?
Why not just run the camera backbone at a lower frame rate and warp the bird's-eye-view features forward? This is the most serious threat. LiDAR keeps metric geometry current every sweep, and the camera's semantic content changes slowly (STTS finds temporal redundancy far exceeds spatial redundancy). Two things argue against it. A frame-level skip cannot spend its compute where change actually is, so newly revealed road and crossing pedestrians wait up to $n$ frames for fresh camera evidence. And BEVFormer's reported gain of more than 6% recall on heavily occluded objects comes from dense per-frame temporal updates, which skipping removes. Whether these effects matter at realistic budgets is settled by our matched-compute comparison, broken down by static, dynamic, and newly visible regions. If frame skipping matches us overall and on dynamic objects, the token-level design is not worth its complexity.
Why not just apply Eventful Transformers as-is? On a moving car, static structure shifts across pixels from frame to frame, so a same-pixel feature difference fires almost everywhere. We expect its recompute rate to approach 100% at the budgets needed for acceptable accuracy, wiping out its savings. This prediction is cheap to check, and we report recompute rate against vehicle speed to show it directly.
Why not just warp the cache in 3D and gate on the residual alone, without any viewing-angle term? This may well be sufficient. The residual is computed on the current patch embedding, which already partly reflects new specular highlights or grazing-angle foreshortening. The case for $\Delta\theta$ is that shallow embeddings may not reveal how much deep features will change. The diagnostic regression and the residual-only ablation decide this. If $\Delta\theta$ adds no explained variance and no NDS at matched budget, we drop it and report the surface-indexed cache as the contribution.
Why not just use a smaller image backbone? A smaller backbone lowers compute uniformly, including on the hard frames where capacity matters. We include a size-matched smaller vision-transformer BEVFusion at equal GFLOPs as an additional reference point.
Proposed Experiments
The first decisive check is diagnostic and comes before any method claims. On full-compute nuScenes runs over the 12 Hz camera sweeps (nuScenes), we warp each token's final-layer feature to the next frames with LiDAR depth. We then measure how feature change grows with token age, viewing-angle change, distance, and surface class. This tells us whether re-observed static surfaces drift meaningfully with angle in driving scenes. Glass facades and wet roads are the cases we expect to break "seeing once."
The main experiment traces accuracy–compute curves for all baselines. We report NDS, mAP and mean velocity error at the annotated keyframes, camera-backbone GFLOPs and wall-clock latency, and per-region recall. The camera stream is processed at full sweep rate so that reuse is exercised between keyframes.
Three stress tests follow. The first uses the night subset, where camera detection degrades sharply according to nuScenes' own analysis; noisier residuals may make geometric cues more valuable there. The second adds injected LiDAR delays of up to 0.5 s, following AsyncBEV's protocol, to measure how stale depth corrupts the warp. The third removes the periodic full refresh to quantify drift.
As a secondary, low-cost test of the viewpoint hypothesis outside driving, we add the angle-aware refresh to the "Seeing Once" pruning pipeline. On ScanQA and SQA3D with Qwen2.5-VL-7B, at matched token count, it should exceed the reported 25.1 and 47.3 Exact Match if angle-dependent re-observation carries usable information.
The proposal is falsified if, at equal compute, the 3D-warped residual-only gate matches the full gate, and frame-rate reduction matches both on dynamic and newly visible regions.
References
- Seeing Once is Enough? Online Geometry-Aware Token Pruning for 3D Question Answering
- Face Recognition: the Problem of Compensating for Changes in Illumination Direction
- Exploiting View-Specific Appearance Similarities Across Classes for Zero-Shot Pose Prediction: A Metric Learning Approach
- Eventful Transformers: Leveraging Temporal Redundancy in Vision Transformers
- MVX-Net: Multimodal VoxelNet for 3D Object Detection
- BEVFusion: A Simple and Robust LiDAR-Camera Fusion Framework
- BEVFormer: Learning Bird's-Eye-View Representation from Multi-camera Images via Spatiotemporal Transformers
- AsyncBEV: Cross-modal flow alignment in Asynchronous 3D Object Detection
- Efficient Video Transformers with Spatial-Temporal Token Selection
- nuScenes: A multimodal dataset for autonomous driving