FPGA-Based Feature Extraction and Tracking Accelerator for Real-Time Visual SLAM

Jie Zhang, Shuai Xiong, Cheng Liu, Yongchao Geng, Wei Xiong, Song Cheng, Fang Hu2023 · DOI

TL;DR

This paper presents an FPGA-based accelerator designed to speed up the front-end processing for Visual SLAM (Simultaneous Localization and Mapping) and Visual Odometry. By combining the FAST feature detector with Gunnar Farneback (GF) dense optical flow and a novel pyramid module, the accelerator achieves real-time performance with significantly reduced latency compared to CPU-based approaches, making it suitable for edge devices.

What Problem Does This Paper Address?

Visual SLAM (V-SLAM) and Visual Odometry (VO) are crucial for location, navigation, and mapping in environments where GPS is unreliable, finding applications in autonomous driving, robotics, and VR. However, these technologies are computationally intensive, requiring high-performance computing platforms that increase costs and limit their widespread adoption. The core challenge lies in processing high-resolution, high-frame-rate image streams quickly and reliably. While GPUs and FPGAs offer hardware acceleration, FPGAs provide inherent advantages in low latency and energy efficiency for streaming tasks due to their reconfigurable nature. Existing FPGA solutions often focus on either feature extraction or optical flow, but rarely combine them efficiently for robust real-time performance, especially when dealing with challenges like scale and rotation invariance.

Methods

The proposed solution is an FPGA-based accelerator that accelerates the image front-end for VO and V-SLAM. Key components include:

  • Image Preprocessing: Utilizes Contrast Limited Adaptive Histogram Equalization (CLAHE) for improved image contrast and Gaussian blur for noise reduction and detail smoothing, preparing the image for downsampling.
  • Pyramid Processing: Implements a five-layer thumbnail structure to address the scale and rotation invariance issues of the FAST feature detector. This module generates downsampled versions of the image.
  • Gunnar Farneback (GF) Dense Optical Flow: This is a core innovation. Unlike sparse optical flow which tracks only keypoints, dense optical flow calculates the displacement of all pixels. This provides more robust tracking and enables dense/semi-dense map construction. The GF method approximates local image neighborhoods with polynomials, allowing for efficient calculation on the FPGA.
  • FAST Feature Extraction: A lightweight algorithm that detects feature points based on brightness differences with surrounding pixels.
  • Feature Tracking and Merging: Combines FAST features with the dense optical flow to track feature points across frames. The system outputs feature point IDs and coordinates to the back-end for pose estimation.

The paper highlights the novel combination of FAST features with GF dense optical flow for improved tracking and the pyramid module for scale/rotation invariance.

Datasets

The paper does not explicitly mention training datasets. For evaluation, it uses a "test benchmark system" that involves acquiring images from a camera and comparing the FPGA's output with MATLAB simulations running on a PC. The "EuRoC dataset" is mentioned in the context of prior work (Navion system), but not used for evaluation in this paper.

Key Insights and Ideas

  • Synergistic Combination of FAST and GF Dense Optical Flow: This is the paper's primary contribution. Combining FAST's efficiency for feature detection with GF dense optical flow's robustness for tracking and mapping offers a powerful solution for real-time V-SLAM, overcoming limitations of using either method in isolation.
  • Pyramid Module for Scale/Rotation Invariance: The five-layer pyramid structure is crucial for making the FAST features more robust to changes in scale and rotation, a common weakness of this detector.
  • FPGA for Low-Latency, High-Throughput Vision Tasks: The work demonstrates the effectiveness of FPGAs in achieving millisecond-level processing latency for complex vision tasks, significantly outperforming CPU-based solutions that take seconds. This makes it ideal for edge computing.
  • Flexibility for Back-end Processing: The accelerator provides output that can be used for pose estimation either by combining FAST features with sparse optical flow or by directly using the dense optical flow for estimation, offering flexibility to the V-SLAM system designer.

Key Results

  • Performance: The accelerator achieves processing speeds of 108 FPS for 720P images and 48 FPS for 1080P images at 100 MHz. It can operate up to 200 MHz.
  • Latency Reduction: Processing latency for 720P images is reduced to approximately 10 ms per frame, a dramatic improvement over PC CPUs which take seconds.
  • Accuracy and Robustness: The accelerator demonstrated stable tracking of features even with violently shaking images and achieved 100% consistency with MATLAB benchmark results.
  • Resource Utilization: On a Xilinx Zynq UltraScale+ MPSoC ZU15EG, the accelerator uses 36% of LUTs, 52% of BRAMs, and 19% of DSPs.

Limitations and Future Work

  • Fixed-Point Computation: The use of fixed-point arithmetic in the FPGA implementation may lead to accuracy issues and overflow problems compared to floating-point computation.
  • Optical Flow Update Accuracy: The limited size of BRAM restricts the optical flow update section to only 8 rows above and below, which can slightly reduce the accuracy and stability of optical flow tracking.
  • Future Work: The paper does not explicitly detail future work, but the limitations suggest potential areas for improvement, such as exploring higher precision arithmetic or optimizing memory access for optical flow updates. The flexibility of the output also opens avenues for integration with various back-end SLAM algorithms.

Summary updated

BibTeX
@misc{zhang2023fpga,
  author = {Jie Zhang and Shuai Xiong and Cheng Liu and Yongchao Geng and Wei Xiong and Song Cheng and Fang Hu},
  title = {{FPGA-Based Feature Extraction and Tracking Accelerator for Real-Time Visual SLAM}},
  year = {2023},
  doi = {10.3390/s23198035},
}

Loading related content...

Figure