Faster than FAST: GPU-Accelerated Frontend for High-Speed VIO

Balázs Nagy, Philipp Foehn, Davide Scaramuzza

Introduction

Visual-Inertial Odometry (VIO) is a critical component for autonomous systems, providing real-time state estimation by fusing camera and IMU data. However, the computational demands of processing high-frequency visual data often lead to significant latency, which can be catastrophic for high-speed drones or lead to motion sickness in AR/VR applications. Researchers from the University of Zurich have addressed this by developing a GPU-accelerated VIO frontend that achieves over 1,000 frames per second (fps) on embedded hardware.

The GPU-accelerated VIO frontend processing layers on the NVIDIA Jetson TX2 platform.

The Latency Bottleneck

While modern sensors can capture data at hundreds of hertz, standard VIO algorithms struggle to maintain such rates on embedded processors. This latency limits the maximum safe speed of autonomous drones and degrades the user experience in wearable headsets. Existing GPU solutions often fail to optimize for the specific bottlenecks of the VIO frontend, such as non-maxima suppression (NMS) and ensuring an even distribution of features across the image.

Re-engineering the VIO Frontend

The proposed system optimizes the frontend by leveraging low-level GPU hardware instructions and rethinking traditional algorithms for parallel execution.

1. Bit-Based FAST Detector

Standard FAST corner detection relies on conditional branching, which causes "execution divergence" on GPUs, forcing threads to wait. The authors replaced this logic with an 8-kilobyte lookup table. By pre-calculating corner patterns, the GPU performs a simple bit-array index lookup, keeping all cores synchronized and maximizing throughput.

2. Warp-Level Non-Maxima Suppression (NMS)

To maintain spatial awareness and feature distribution, the image is divided into a grid. The system uses warp-level shuffle instructions—a feature of the NVIDIA Kepler architecture and newer—to allow threads within a warp to exchange data directly. This enables the simultaneous identification of the strongest features in each grid cell without accessing slow global memory.

Generalized logic for feature detection across scales and regions of interest.

3. Collaborative Feature Tracking

Rather than the traditional one-thread-per-feature approach, which results in inefficient memory access patterns, this system assigns an entire warp (32 threads) to each feature patch. This collaborative method ensures memory requests are coalesced, significantly increasing effective bandwidth.

Experimental Results

The system was benchmarked using the EuRoC MAV dataset, a standard for drone-based state estimation.

  • Throughput: The standalone frontend reached over 1,000 fps on the NVIDIA Jetson TX2. When integrated into a full VIO pipeline (with ICE-BA), it maintained a rate of ~200 fps.
  • Efficiency: The novel NMS scheme provided a 2x speedup over previous state-of-the-art methods (such as Forstner’s spiral NMS) on embedded hardware.
  • Accuracy: Translation error remained comparable to CPU-based implementations, typically around 1% of the total trajectory length, proving that speed does not come at the cost of precision.

Conclusion and Future Outlook

By optimizing for the "GPU's perspective," this work demonstrates that embedded hardware can handle ultra-high-speed VIO. While currently optimized for CUDA-enabled NVIDIA hardware, future developments could extend these techniques to OpenCL or Vulkan. Addressing challenges like motion blur in low-light environments remains an area for future research, but the current throughput gains represent a major leap forward for high-speed autonomous flight.

Summary updated

BibTeX
@inproceedings{nagy2020faster,
  author = {Balázs Nagy and Philipp Foehn and Davide Scaramuzza},
  title = {{Faster than FAST: GPU-Accelerated Frontend for High-Speed VIO}},
  booktitle = {IEEE/RSJ International Conference on Intelligent Robots and Systems},
  year = {2020},
  doi = {10.1109/iros45743.2020.9340851},
}
Figure