TL;DR
The S$^2$D-FVO framework solves the persistent problem of scale drift in monocular fisheye visual odometry by coupling a novel spherical bundle adjustment (SBA) with unsupervised depth learning. By minimizing angular residuals on a unit sphere and using a median-depth scale constraint, the system achieves state-of-the-art accuracy in both trajectory tracking and depth estimation across diverse real-world and synthetic environments.
What Problem Does This Paper Address?
Monocular Visual Odometry (VO)—estimating a camera's path from a single video stream—is a cornerstone of robotic navigation. Fisheye cameras are particularly attractive for this because their wide Field of View (FoV) ensures significant visual overlap between frames, leading to more stable tracking.
However, two major hurdles remain:
- Scale Ambiguity: A single camera cannot inherently distinguish between moving a short distance in a small scene versus a long distance in a large one. Over time, small errors accumulate into "scale drift," causing the estimated trajectory to warp.
- Extreme Distortion: Standard VO algorithms rely on "pixel reprojection error." In fisheye images, a one-pixel error at the center represents a much smaller angular deviation than a one-pixel error at the highly compressed edges. This non-uniformity makes traditional optimization (Bundle Adjustment) unstable and inaccurate.
While unsupervised depth learning can provide a scale reference, training these models on fisheye data is difficult because the "PoseNets" typically used to provide supervision struggle with the complex motion patterns caused by fisheye distortion.
Methods
The authors introduce S$^2$D-FVO, which features several key innovations:
- Equirectangular Projection (ERP) Remapping: To handle various fisheye models (e.g., Kannala-Brandt, Scaramuzza) uniformly, raw images are remapped to the ERP domain. This preserves spatial continuity better than "cubemap" remapping, which can introduce discontinuities at edges.
- Spherical Bundle Adjustment (SBA): Instead of minimizing pixel distances, SBA operates on the unit sphere. It minimizes the angular deviation between observed and predicted viewing rays. To make this mathematically robust for optimization, the authors use a spherical logarithmic map to project residuals into a 2D tangent space. This ensures that an angular error of 1° has the same "weight" regardless of where it appears in the image.
- SBA-Guided Unsupervised Learning: During the training of the depth estimation network (DepthNet), the authors replace the standard black-box PoseNet with their explicit SBA algorithm. By jointly refining relative poses and 3D landmarks geometrically, they provide much more reliable "photometric supervision" for the DepthNet.
- Median Depth Scale Regularization: In the final VO pipeline, the trained DepthNet provides a scale reference. Rather than forcing every point to match the network's prediction (which might be noisy), the system enforces consistency between the median depth of reconstructed landmarks and the median predicted depth. This anchors the global scale while allowing the geometry to refine individual points.
Datasets
The framework was evaluated on a mix of public and newly created benchmarks:
- Public: KITTI-360 (suburban driving), TUM-VI (handheld indoor), ScanNet++ (indoor laser-scanned), and TartanAir V2 (large-scale synthetic).
- New - NIO-Fisheye: A real-world driving dataset collected with an NIO ES8 vehicle equipped with high-resolution fisheye cameras and LiDAR for ground truth.
- New - MIAS-FisheyeVO: A high-quality synthetic dataset rendered in Blender, featuring complex indoor and outdoor trajectories.
Key Insights and Ideas
- Geometry over Black-Boxes: The "aha moment" is that for fisheye cameras, explicit geometric optimization (SBA) is a far better teacher for a neural network than another neural network (PoseNet). The geometric constraints of the unit sphere are rigid and reliable, whereas PoseNets often fail to generalize to extreme distortions.
- The Unit Sphere is the "Fair" Domain: By moving optimization from the distorted image plane to the unit sphere, the authors effectively "de-distort" the math of visual odometry without losing the benefits of the wide FoV.
- Median Anchoring: Using the median depth for scale regularization is a clever way to handle the inherent noisiness of monocular depth networks. It provides a stable "anchor" for the world's size without letting a few bad depth predictions ruin the entire reconstruction.
Key Results
The proposed framework significantly outperformed existing geometry-based and learning-based methods:
- Trajectory Accuracy: On the KITTI-360 dataset, S$^2$D-FVO reduced translational error by 92.3% and rotational error by 95.8% compared to the geometry-based baseline OV$^2$SLAM.
- Depth Estimation: It achieved state-of-the-art results among unsupervised methods, reducing the Mean Absolute Relative Error (Abs Rel) by up to 57.4% on the ScanNet++ dataset compared to perspective-based models.
- Efficiency: Despite the complex optimization, the system runs at approximately 10 FPS on a modern workstation (Intel Core Ultra 9 + RTX 5090), making it suitable for near-real-time robotic applications.
Limitations and Future Work
While highly effective, the authors note two main limitations:
- Systematic Bias: If the DepthNet has a systematic bias (e.g., consistently overestimating distances in a specific type of tunnel), the median-based scale reference will inherit that bias.
- Parallax Dependence: Like all triangulation-based methods, the system's ability to determine depth (and thus scale) relies on camera motion. In cases of pure rotation or very slow movement, the geometric constraints weaken.
Future work will focus on adaptive scale regularization—developing ways for the system to automatically judge how much it should trust the DepthNet versus the geometric triangulation at any given moment.
Summary updated