Introduction
Transformers have become a cornerstone of computer vision, including 3D object detection for autonomous driving. However, their self-attention mechanism's computational complexity grows quadratically with the number of input tokens. In 3D LiDAR scenes, the vast majority of "tokens" (voxels) represent empty space or irrelevant background, yet standard transformers process all tokens with equal effort.
Existing "token pruning" methods for image classification often rely on a single global class token, which is unsuitable for detection where objects can appear anywhere. Furthermore, autonomous driving requires deterministic behavior for safety, making stochastic halting methods a liability. This paper introduces a framework to accelerate transformer-based 3D detectors by dynamically "halting" unimportant tokens at different network layers.
Methodology
The proposed framework is built upon the Single-stride Sparse Transformer (SST) and introduces several key innovations to manage token lifecycles efficiently:
- Dynamic Halting Module: Lightweight modules assign importance scores to tokens at specific layers. Tokens with scores below a threshold are "halted," bypassing subsequent expensive transformer layers.
- Token Recycling: To preserve spatial context, halted tokens are not discarded. Instead, their features are forwarded directly to the final detection head, ensuring background voxels still contribute to the bird's-eye-view (BEV) map.
- Weighted Self-Attention (WSA): Halting scores weight the self-attention mechanism, encouraging the model to focus computational resources on tokens likely to represent objects.
- Equivalent Differentiable Forward-pass (EDF): To overcome the non-differentiability of discrete halting, a training-time proxy and Straight-Through Estimator (STE) are used to enable end-to-end gradient flow.
- Non-Uniform Sparsity Loss: A heatmap-based loss, derived from ground-truth bounding boxes, penalizes the model for halting tokens near actual objects.
Results and Discussion
The method was evaluated on the Waymo Open Dataset (WOD). A primary finding is that halting background tokens actually improves the signal-to-noise ratio, allowing the model to focus on sparse or distant objects.
Efficiency and Accuracy
The framework achieves a 50% speedup in the transformer backbone with negligible accuracy loss. By reinvesting these computational savings into a more complex detection head (e.g., adding a Feature Pyramid Network), the resulting model, $SST^{++}_{halt}$, outperforms the state-of-the-art SST by 1.0% to 3.5% AP/APH across various categories without increasing total latency.
Performance Breakdown
The model showed significant gains in long-range detection ($[50m, \infty)$), suggesting that dynamic halting effectively addresses the "needle in a haystack" problem of identifying distant pedestrians and vehicles amidst background noise.
Conclusion
This work demonstrates that dynamic, deterministic token halting is a viable strategy for making 3D transformers more efficient for real-time autonomous driving. By recycling halted tokens and using non-uniform sparsity losses, the model maintains high detection accuracy while significantly reducing latency. Future work includes extending this logic to multi-modal (Camera-LiDAR) systems and exploring temporal token halting across multiple frames.
Summary updated