Towards Real-Time Full-Waveform LiDAR Transformers via Intensity-Guided Token Reduction and Physics-Aware Augmentation

Kotaro Oishi, Kazuma Ikeda, Ryosei Hara, Ryo Yoshida, Mariko Isogawa, Kentaro YoshiokaarXiv 2026 · arXiv

TL;DR

This paper introduces the first real-time Transformer for Full-Waveform LiDAR (FWL), achieving an 8× speedup (53.2 FPS) and a significant 9-point F1 accuracy gain over previous state-of-the-art models. By strategically pruning 70% of uninformative "noise" tokens and using physics-aware data augmentation, the authors prove that throwing away the right data can actually make a model both faster and smarter.

What Problem Does This Paper Address?

Conventional LiDAR is the backbone of autonomous driving, but it has a "vision" problem: it usually only records the strongest return signal for each laser pulse. This makes it struggle with transparent objects (glass), low-reflectance surfaces, and adverse weather like fog or rain, which create "ghost" points or noise.

Full-waveform LiDAR (FWL) solves this by recording the entire temporal intensity histogram (the waveform) for every pixel. While this provides a rich dataset capable of seeing through fog or identifying glass, it creates a massive computational bottleneck. The data volume is enormous, yet most of it consists of low-intensity background noise. Existing Transformer models for FWL are too slow for real-time use (often <7 FPS) and struggle to generalize to new environments because annotated FWL data is extremely scarce.

Methods

The authors propose two complementary frameworks to tackle efficiency and generalization:

1. FWL-ToPM (Token Pruning and Merging)

Instead of processing every voxel, FWL-ToPM exploits the inherent sparsity of LiDAR data:

  • Intensity-Based Pruning: The model evaluates the maximum signal intensity of 3D patches and discards the bottom 70%. These regions are typically just empty space or sensor noise.
  • Sparse-Aware Merging: For the remaining "important" tokens, the model uses a space-filling curve (specifically Morton order) to map the sparse 3D grid into a 1D sequence while preserving spatial proximity. It then merges similar tokens to further reduce the workload for the Transformer's self-attention mechanism.
Figure 1: Overview of the proposed framework. (Top) During training, FWLAug augments FWL samples while preserving ToF waveform structure and frustum geometry. High-intensity patches are selected by pruning, spatially reordered along a space-filling curve, and processed by the FWL-ToPM Encoder, which merges redundant tokens before self-attention and unmerges afterward across L Transformer blocks. (Bottom Left) Pruning removes low-intensity background tokens, and merging aggregates the remainder, concentrating computation on object-relevant waveform content. (Bottom Right) Proportion of pixel classes across intensity bins in FWL data. Most regions in FWL data correspond to noisy background, and our pruning strategy removes these dominant low-intensity regions.

2. FWLAug (Physics-Aware Augmentation)

Standard image augmentations (like cropping or flipping) break the physical relationship between the time-of-flight (ToF) and the 3D geometry of a LiDAR scan. FWLAug introduces two new methods:

  • FWLShiftMix: Translates objects in 3D space and recomputes their waveforms to match their new distances, filling the resulting "holes" with data from other scenes.
  • FWLCutMix: Pastes regions containing minority classes (like glass) into new frames to help the model learn rare reflection signatures.
Figure 2: Illustration of FWLAug. FWLShiftMix (top): an object in Sample 1 is shifted away from the LiDAR, creating a blank region Ωblank. A glass-containing region Ωsrc is then selected from Sample 2 and pasted into the blank, producing an augmented scene in which the waveform structure of each ray is preserved. FWLCutMix (bottom): a crop region Ωsrc containing minority class (glass) is selected from a source frame and pasted into a destination frame at a random location. Both methods prioritize glass-containing regions to mitigate class imbalance.

Datasets

  • Ghost-FWL: The primary benchmark for FWL-based ghost detection, containing peak-level annotations for noise, objects, glass, and ghosts.
  • OOD Test Set: The authors introduced a new Out-of-Distribution (OOD) dataset captured at a completely different location (a glass-heavy seven-story building) to test how well the models generalize to unseen environments.

Key Insights and Ideas

  • Pruning as Regularization: A surprising finding was that aggressive pruning (up to 70%) didn't just save time—it improved accuracy. In FWL, low-intensity tokens act as "distractors." Removing them allows the Transformer to focus exclusively on object-relevant signals.
  • Geometry Matters: You cannot treat FWL data like a simple 3D tensor. Because each pixel represents a ray originating from a single point (the sensor), any augmentation must respect this "frustum geometry" and the temporal structure of the waveform to remain physically plausible.
  • Serialization is the Key to Sparsity: By using space-filling curves, the authors found a way to apply dense Transformer operations to sparse, irregular data without losing the spatial context that makes merging effective.

Key Results

The proposed framework, FWL-ToPM, sets a new bar for the field:

  • Speed: Achieved 53.2 FPS, well above the 10 Hz requirement for real-time autonomous systems, compared to the previous SOTA's 6.5 FPS.
  • Accuracy: Improved the F1-mean score from 0.503 to 0.592 on the Ghost-FWL benchmark.
  • Generalization: On the new OOD test set, the combination of FWL-ToPM and FWLAug significantly outperformed all baselines, particularly in detecting the difficult "Glass" class.
F1-mean vs. FPS trade-off showing the superiority of the proposed method.

Limitations and Future Work

  • Weak Signal Risk: The current pruning strategy relies on intensity. There is a risk that very weak but important signals (e.g., from highly distant or low-reflectance objects) might be accidentally pruned. The authors suggest exploring adaptive pruning thresholds in the future.
  • Sensor Specificity: The evaluation was limited to a single FWL sensor platform. Cross-sensor validation remains an open challenge.
  • System Latency: While the model is fast, the authors note that a full deployment would need to account for end-to-end system latency, including sensor data transfer and preprocessing.

Summary updated

Abstract

LiDAR is a critical sensor for autonomous driving and robotics, yet it remains vulnerable to adverse conditions such as fog, rain, transparent objects, and low-reflectance surfaces. Full-waveform LiDAR (FWL) addresses these limitations by capturing the complete return waveform as a histogram, preserving rich reflection characteristics including multi-path echoes and weak signals. While recent Transformer-based approaches have demonstrated promise on FWL data, two fundamental bottlenecks remain: high computational cost from redundant tokens in low-intensity background regions, and limited generalization due to scarce annotated training data. We address both bottlenecks by exploiting the physical structure inherent to FWL data. First, we propose FWL-ToPM, a token pruning and merging mechanism that leverages intensity distributions to selectively reduce tokens while retaining critical waveform content. Second, we introduce FWLAug, a data augmentation framework designed as a regularization method that preserves waveform structure, retaining the Time-of-Flight waveform of each ray and the frustum geometry, to improve generalization to unseen scenes. Evaluated on the Ghost-FWL benchmark, FWL-ToPM achieves over 8 times speedup at real-time throughput alongside a nearly 9-point F1 gain over the prior state of the art, establishing the first real-time FWL Transformer that simultaneously improves accuracy. FWLAug further improves performance on both the standard benchmark and an out-of-distribution test set of unseen scenes. Together, these methods push FWL-based perception toward practical deployment. The code and the OOD dataset are publicly available and can be accessed via the project page: https://keio-csg.github.io/FWL-ToPM
BibTeX
@misc{oishi2026towards,
  author = {Kotaro Oishi and Kazuma Ikeda and Ryosei Hara and Ryo Yoshida and Mariko Isogawa and Kentaro Yoshioka},
  title = {{Towards Real-Time Full-Waveform LiDAR Transformers via Intensity-Guided Token Reduction and Physics-Aware Augmentation}},
  year = {2026},
  eprint = {2611.07613},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.07613},
}
Figure