A Tightly Coupled Visual-Inertial GNSS State Estimator Based on Point-Line Feature

Kai Zhang, Bo Dong, Kai Zhang 0012

TL;DR

This paper introduces a novel tightly coupled GNSS-Visual-Inertial Odometry (VIO) system that leverages both point and line features for robust state estimation. By fusing carrier-phase smoothed pseudorange GNSS measurements with visual and inertial data, and enabling real-time extrinsic parameter calibration, the system achieves high-precision, drift-free localization in challenging environments.

What Problem Does This Paper Address?

Accurate and robust localization is crucial for applications like robotics, autonomous driving, and UAVs. While Visual Odometry (VO) offers low-cost solutions, it suffers from drift and struggles in environments with weak or repetitive textures. Inertial Measurement Units (IMUs) provide high-frequency data but also drift over time. Global Navigation Satellite Systems (GNSS) offer global positioning but standard pseudorange measurements have meter-level accuracy and are susceptible to errors. Carrier phase GNSS offers centimeter-level accuracy but requires complex ambiguity resolution, hindering real-time performance. Existing tightly coupled GNSS-VIO systems aim to fuse these modalities, but often struggle with feature robustness in challenging visual conditions and the trade-offs between GNSS measurement types. This paper addresses these limitations by proposing a system that enhances feature robustness, improves GNSS measurement accuracy and real-time usability, and ensures full state observability in the ECEF frame.

Methods

The proposed system employs a tightly coupled approach, integrating measurements from cameras, IMUs, and GNSS receivers. Key methodological contributions include:

  • Point-Line Feature Fusion: To overcome the limitations of point features in texture-poor environments, the system extracts and utilizes line features alongside traditional point features. This provides richer structural information for more robust tracking and mapping.
  • Carrier Phase Smoothed Pseudorange: Instead of directly using GNSS pseudorange measurements (which are fast but inaccurate) or carrier phase measurements (which are accurate but slow to resolve ambiguity), the system uses carrier phase smoothed pseudorange. This technique, often implemented with a Hatch filter, leverages the carrier phase to improve the accuracy of pseudorange measurements while maintaining real-time performance.
  • Real-time GNSS-IMU Extrinsic Calibration: The spatial offset between the GNSS receiver and IMU is critical for accurate fusion. The proposed system can calibrate this extrinsic translation parameter in real-time, eliminating the need for offline calibration.
  • ECEF Frame Observability: The system demonstrates that by representing states in the Earth-Centered, Earth-Fixed (ECEF) frame, full observability is achieved, unlike VIO systems that often have unobservable directions in local frames. This is crucial for eliminating accumulated drift over long trajectories.
  • Factor Graph Optimization: The system is formulated as a nonlinear optimization problem on a factor graph, incorporating factors for point features, line features, IMU preintegration, smoothed pseudorange, and Doppler frequency shift.

Datasets

The experiments were conducted on two public datasets:

  • GVINS-Dataset: This dataset captures scenes from the Hong Kong University of Science and Technology, including a sports field and a complex indoor-outdoor environment. It was collected using a helmet-mounted VI-sensor and a u-blox ZED-F9P GNSS receiver. The sports field dataset is characterized by an open environment with good GNSS reception, while the complex environment dataset includes challenging scenarios like dim lighting, occlusions, and GNSS signal corruption.

Key Insights and Ideas

  • Line features are crucial for robustness: In environments with challenging visual conditions (e.g., weak textures, repetitive patterns), line features provide essential structural information that point features alone cannot capture, leading to more reliable state estimation.
  • Carrier phase smoothing is a practical GNSS enhancement: By combining the speed of pseudorange with the accuracy of carrier phase, carrier phase smoothed pseudorange offers a significant improvement in positioning precision without the computational burden of full carrier phase ambiguity resolution.
  • ECEF frame observability is key to drift-free localization: Representing states in the global ECEF frame, rather than a local frame, unlocks full observability, enabling the system to correct for accumulated drift and provide globally consistent trajectories.
  • Real-time extrinsic calibration simplifies deployment: The ability to calibrate the GNSS-IMU extrinsic parameters online removes a significant barrier to practical implementation, making the system more adaptable to different sensor setups.

Key Results

  • Sports Field: The proposed system achieved drift-free trajectories in the sports field, with positioning errors generally less than 1m in the ENU frame. Its performance was comparable to or better than GVINS, a state-of-the-art GNSS-VIO system, particularly in terms of RMSE and SEP in the ECEF frame.
  • Complex Environment: In challenging environments with GNSS occlusions and poor visual features, the system demonstrated robustness. While VINS-Mono showed significant drift, the proposed system and GVINS maintained accurate trajectories. The system's positioning error was larger than in the sports field but still effective, with the SEP metric indicating better performance than GVINS on this dataset.
  • Smoothed Interval Sensitivity: Experiments showed that an optimal smoothed interval (e.g., 5-50 epochs) is necessary for carrier phase smoothed pseudorange. Too large an interval can lead to issues with cycle slips, while too small an interval can degrade performance.
  • Line Feature Tracking Threshold: A threshold of 3-4 consecutive keyframes for line feature observation yielded more accurate results, indicating that stable line features are crucial for robust optimization.

Limitations and Future Work

  • Limitations: The paper acknowledges that in extremely challenging scenarios with severe GNSS signal loss or complete lack of visual features, the system's performance might be affected. The performance of the carrier phase smoothed pseudorange is dependent on continuous carrier phase lock.
  • Future Work: The authors suggest focusing on further improvements for tightly coupled GNSS-VIO systems, implying potential areas for future research could include more advanced feature extraction techniques, improved GNSS signal processing, and enhanced robustness in extreme conditions.

Summary updated

Abstract

Visual-inertial odometry (VIO) is known to suffer from drifting and can only provide local coordinates. In this paper, we propose a tightly coupled GNSS-VIO system based on point-line features for robust and drift-free state estimation. Feature-based methods are not robust in complex areas such as weak or repeated textures. To deal with this problem, line features with more environmental structure information can be extracted. In addition, to eliminate the accumulated drift of VIO, we tightly fused the GNSS measurement with visual and inertial information. The GNSS pseudorange measurements are real-time and unambiguous but experience large errors. The GNSS carrier phase measurements can achieve centimeter-level positioning accuracy, but the solution to the whole-cycle ambiguity is complex and time-consuming, which degrades the real-time performance of a state estimator. To combine the advantages of the two measurements, we use the carrier phase smoothed pseudorange instead of pseudorange to perform state estimation. Furthermore, the existence of the GNSS receiver and IMU also makes the extrinsic parameter calibration crucial. Our proposed system can calibrate the extrinsic translation parameter between the GNSS receiver and IMU in real-time. Finally, we show that the states represented in the ECEF frame are fully observable, and the tightly coupled GNSS-VIO state estimator is consistent. We conducted experiments on public datasets. The experimental results demonstrate that the positioning precision of our system is improved and the system is robust and real-time.
BibTeX
@article{zhang2022tightly,
  author = {Kai Zhang and Bo Dong and Kai Zhang},
  title = {{A Tightly Coupled Visual-Inertial GNSS State Estimator Based on Point-Line Feature}},
  journal = {Sensors},
  year = {2022},
  doi = {10.3390/s22093391},
}
Figure