Efficient Transformer-based 3D Object Detection with Dynamic Token Halting

Mao Ye, Gregory P. Meyer, Yuning Chai, Qiang LiuICCV 2023 · DOI · OpenReview

Introduction

Transformers have become a cornerstone of computer vision, including 3D object detection for autonomous driving. However, their self-attention mechanism's computational complexity grows quadratically with the number of input tokens. In 3D LiDAR scenes, the vast majority of "tokens" (voxels) represent empty space or irrelevant background, yet standard transformers process all tokens with equal effort.

Existing "token pruning" methods for image classification often rely on a single global class token, which is unsuitable for detection where objects can appear anywhere. Furthermore, autonomous driving requires deterministic behavior for safety, making stochastic halting methods a liability. This paper introduces a framework to accelerate transformer-based 3D detectors by dynamically "halting" unimportant tokens at different network layers.

Methodology

The proposed framework is built upon the Single-stride Sparse Transformer (SST) and introduces several key innovations to manage token lifecycles efficiently:

  1. Dynamic Halting Module: Lightweight modules assign importance scores to tokens at specific layers. Tokens with scores below a threshold are "halted," bypassing subsequent expensive transformer layers.
  2. Token Recycling: To preserve spatial context, halted tokens are not discarded. Instead, their features are forwarded directly to the final detection head, ensuring background voxels still contribute to the bird's-eye-view (BEV) map.
  3. Weighted Self-Attention (WSA): Halting scores weight the self-attention mechanism, encouraging the model to focus computational resources on tokens likely to represent objects.
  4. Equivalent Differentiable Forward-pass (EDF): To overcome the non-differentiability of discrete halting, a training-time proxy and Straight-Through Estimator (STE) are used to enable end-to-end gradient flow.
  5. Non-Uniform Sparsity Loss: A heatmap-based loss, derived from ground-truth bounding boxes, penalizes the model for halting tokens near actual objects.
The proposed architecture featuring dynamic halting, token recycling, and weighted self-attention.

Results and Discussion

The method was evaluated on the Waymo Open Dataset (WOD). A primary finding is that halting background tokens actually improves the signal-to-noise ratio, allowing the model to focus on sparse or distant objects.

Efficiency and Accuracy

The framework achieves a 50% speedup in the transformer backbone with negligible accuracy loss. By reinvesting these computational savings into a more complex detection head (e.g., adding a Feature Pyramid Network), the resulting model, $SST^{++}_{halt}$, outperforms the state-of-the-art SST by 1.0% to 3.5% AP/APH across various categories without increasing total latency.

mAP vs. Speedup comparison showing the superior efficiency-accuracy trade-off of the proposed SST_halt compared to standard scaling methods.

Performance Breakdown

The model showed significant gains in long-range detection ($[50m, \infty)$), suggesting that dynamic halting effectively addresses the "needle in a haystack" problem of identifying distant pedestrians and vehicles amidst background noise.

Conclusion

This work demonstrates that dynamic, deterministic token halting is a viable strategy for making 3D transformers more efficient for real-time autonomous driving. By recycling halted tokens and using non-uniform sparsity losses, the model maintains high detection accuracy while significantly reducing latency. Future work includes extending this logic to multi-modal (Camera-LiDAR) systems and exploring temporal token halting across multiple frames.

Summary updated

Abstract

Balancing efficiency and accuracy is a long-standing problem for deploying deep learning models. The trade-off is even more important for real-time safety-critical systems like autonomous vehicles. In this paper, we propose an effective approach for accelerating transformer-based 3D object detectors by dynamically halting tokens at different layers depending on their contribution to the detection task. Although halting a token is a non-differentiable operation, our method allows for differentiable end-to-end learning by leveraging an equivalent differentiable forward-pass. Furthermore, our framework allows halted tokens to be reused to inform the model's predictions through a straightforward token recycling mechanism. Our method significantly improves the Pareto frontier of efficiency versus accuracy when compared with the existing approaches. By halting tokens and increasing model capacity, we are able to improve the baseline model's performance without increasing the model's latency on the Waymo Open Dataset.
BibTeX
@inproceedings{ye2023efficient,
  author = {Mao Ye and Gregory P. Meyer and Yuning Chai and Qiang Liu},
  title = {{Efficient Transformer-based 3D Object Detection with Dynamic Token Halting}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2023},
  doi = {10.1109/iccv51070.2023.00775},
}
Figure