TL;DR
This paper introduces the first real-time Transformer for Full-Waveform LiDAR (FWL), achieving an 8× speedup (53.2 FPS) and a significant 9-point F1 accuracy gain over previous state-of-the-art models. By strategically pruning 70% of uninformative "noise" tokens and using physics-aware data augmentation, the authors prove that throwing away the right data can actually make a model both faster and smarter.
What Problem Does This Paper Address?
Conventional LiDAR is the backbone of autonomous driving, but it has a "vision" problem: it usually only records the strongest return signal for each laser pulse. This makes it struggle with transparent objects (glass), low-reflectance surfaces, and adverse weather like fog or rain, which create "ghost" points or noise.
Full-waveform LiDAR (FWL) solves this by recording the entire temporal intensity histogram (the waveform) for every pixel. While this provides a rich dataset capable of seeing through fog or identifying glass, it creates a massive computational bottleneck. The data volume is enormous, yet most of it consists of low-intensity background noise. Existing Transformer models for FWL are too slow for real-time use (often <7 FPS) and struggle to generalize to new environments because annotated FWL data is extremely scarce.
Methods
The authors propose two complementary frameworks to tackle efficiency and generalization:
1. FWL-ToPM (Token Pruning and Merging)
Instead of processing every voxel, FWL-ToPM exploits the inherent sparsity of LiDAR data:
- Intensity-Based Pruning: The model evaluates the maximum signal intensity of 3D patches and discards the bottom 70%. These regions are typically just empty space or sensor noise.
- Sparse-Aware Merging: For the remaining "important" tokens, the model uses a space-filling curve (specifically Morton order) to map the sparse 3D grid into a 1D sequence while preserving spatial proximity. It then merges similar tokens to further reduce the workload for the Transformer's self-attention mechanism.
2. FWLAug (Physics-Aware Augmentation)
Standard image augmentations (like cropping or flipping) break the physical relationship between the time-of-flight (ToF) and the 3D geometry of a LiDAR scan. FWLAug introduces two new methods:
- FWLShiftMix: Translates objects in 3D space and recomputes their waveforms to match their new distances, filling the resulting "holes" with data from other scenes.
- FWLCutMix: Pastes regions containing minority classes (like glass) into new frames to help the model learn rare reflection signatures.
Datasets
- Ghost-FWL: The primary benchmark for FWL-based ghost detection, containing peak-level annotations for noise, objects, glass, and ghosts.
- OOD Test Set: The authors introduced a new Out-of-Distribution (OOD) dataset captured at a completely different location (a glass-heavy seven-story building) to test how well the models generalize to unseen environments.
Key Insights and Ideas
- Pruning as Regularization: A surprising finding was that aggressive pruning (up to 70%) didn't just save time—it improved accuracy. In FWL, low-intensity tokens act as "distractors." Removing them allows the Transformer to focus exclusively on object-relevant signals.
- Geometry Matters: You cannot treat FWL data like a simple 3D tensor. Because each pixel represents a ray originating from a single point (the sensor), any augmentation must respect this "frustum geometry" and the temporal structure of the waveform to remain physically plausible.
- Serialization is the Key to Sparsity: By using space-filling curves, the authors found a way to apply dense Transformer operations to sparse, irregular data without losing the spatial context that makes merging effective.
Key Results
The proposed framework, FWL-ToPM, sets a new bar for the field:
- Speed: Achieved 53.2 FPS, well above the 10 Hz requirement for real-time autonomous systems, compared to the previous SOTA's 6.5 FPS.
- Accuracy: Improved the F1-mean score from 0.503 to 0.592 on the Ghost-FWL benchmark.
- Generalization: On the new OOD test set, the combination of FWL-ToPM and FWLAug significantly outperformed all baselines, particularly in detecting the difficult "Glass" class.
Limitations and Future Work
- Weak Signal Risk: The current pruning strategy relies on intensity. There is a risk that very weak but important signals (e.g., from highly distant or low-reflectance objects) might be accidentally pruned. The authors suggest exploring adaptive pruning thresholds in the future.
- Sensor Specificity: The evaluation was limited to a single FWL sensor platform. Cross-sensor validation remains an open challenge.
- System Latency: While the model is fast, the authors note that a full deployment would need to account for end-to-end system latency, including sensor data transfer and preprocessing.
Summary updated