Flexible Trinocular: Non-rigid Multi-Camera-IMU Dense Reconstruction for UAV Navigation and Mapping

Timo Hinzmann, Cesar Cadena, Juan Nieto, Roland SiegwartarXiv 2019 · arXiv

TL;DR

Fixed-wing UAVs require wide stereo baselines for long-range mapping, but flexible wings dynamically alter camera geometry during flight. This paper introduces a flexible trinocular visual-inertial framework that estimates time-varying camera baselines in real-time by fusing relative inertial measurements, a probabilistic wing model, and direct photometric alignment.

What Problem Does This Paper Address?

For a fixed-wing drone traveling at 15 m/s, standard sensing solutions fall short. Monocular depth-from-motion suffers from high uncertainty along the flight path, while sensors like LiDAR or radar are often too heavy, expensive, or power-hungry for small platforms.

Stereo vision is a compelling alternative, but it faces a geometric trap. A small, rigid stereo rig confined to the drone's fuselage (approx. 12 cm baseline) suffers from massive depth uncertainty at longer ranges. To see far enough ahead to safely navigate, the cameras must be placed further apart—such as on the wingtips, expanding the baseline to 2.4 meters.

However, aircraft wings flex and vibrate dynamically due to aerodynamic lift and turbulence. Because standard stereo algorithms assume a perfectly rigid, pre-calibrated transformation between cameras, even slight wing bending completely invalidates the geometric assumptions required for depth calculation.

Fig. 2: Theoretical depth error and disparity for different stereo baselines in meters. The red vertical line marks the distance that our UAV platform can cover within 5 s. Further parameters: focal length of 2.8 mm, pixel size of 6 µm, assumed resolvable disparity of one pixel.

Methods

The authors develop a Flexible Trinocular system consisting of a center camera (rigidly attached to the fuselage autopilot) and two wingtip cameras. To track the non-rigid baseline changes in real-time, they implement an Extended Kalman Filter (EKF) that fuses three distinct components:

  1. Relative Inertial Sensing: High-frequency IMUs mounted behind each camera measure relative linear accelerations and angular velocities.
  2. Probabilistic Wing Model: A Gaussian prior representing the expected wing deformation under aerodynamic load. This model is calibrated in-air by using additional fuselage-mounted side cameras that track AprilTags on the wingtips during a baseline flight.
  3. Direct Photometric Update: Instead of tracking individual sparse features across frames (which is brittle under wing vibration), the system projects features from the center camera into the wing cameras using the EKF's prior estimate. It then refines the full-scale relative camera pose directly by minimizing the photometric (intensity) error between image patches.

Optimized camera poses are then passed to a standard rectification and stereo block-matching pipeline to generate dense depth maps.

Fig. 3: Overview showing the UAV platform, sensors, and coordinate frames. An EKF estimates the relative pose between left, right, and center camera. The side cameras are used to identify a wing model in form of a Gaussian prior. The horizontal field of view (HFoV) is denoted for each camera. The schematic on the right bottom depicts the developed VI-sensor that time-synchronizes the sensors on a hardware level.

Datasets

The framework was evaluated using real-world flight data collected via a customized Techpod fixed-wing UAV platform. The drone was outfitted with a custom hardware-synchronized visual-inertial sensor array, including Aptina MT9V034 cameras and MPU6050/ADIS16448 IMUs, managed by an on-board Intel Atom computer. No standard public datasets were used, as the study required specialized non-rigid hardware configurations.

Key Insights and Ideas

  • Ground Calibration is Insufficient: Calibrating the camera system on the ground yields poor results because it fails to account for the constant aerodynamic lift forces that bend the wings upward during flight.
  • Direct Alignment Overcomes Feature Matching Failures: Traditional feature tracking (like Lucas-Kanade) across a flexing baseline produces high outlier rates due to changing illumination and large search spaces. Constraining the search space via a relative transformation prior and optimizing via direct photometric alignment proves far more robust.
  • Vibration is a Software Problem: Wingtips experience violent high-frequency vibrations. Rather than relying purely on mechanical damping, the authors successfully mitigated this by applying an aggressive on-board digital low-pass filter (5 Hz cut-off) directly to the wing IMUs.

Key Results

  • Dynamic Baseline Tracking: The system successfully tracked real-time wing deflections, showing that the relative translation and orientation varied by up to 5 cm and 6° respectively between taxiing and level flight.
  • High Efficiency: The state estimation loop is highly optimized. Relative IMU integration takes just 0.03 ms, and the photometric refinement requires only 3.2 ms per frame, easily supporting real-time 10 Hz operation.
  • Reconstruction Bottleneck: While the pose estimation is lightweight, the dense stereo block-matching step takes 38.1 ms per stereo pair, identifying it as the primary computational bottleneck preventing higher frame rates.
Fig. 1: Fixed-wing UAV platform Techpod equipped with the proposed trinocular visual-inertial sensor setup. The side cameras and April tags are used for the identification of the wing model.

Limitations and Future Work

While effective, the current wing model is a simple static Gaussian prior. The authors note that a more sophisticated structural model—such as treating the wing as a cantilever beam and incorporating real-time airspeed data—could yield higher accuracy. Additionally, the violent high-frequency vibrations are currently handled in software; introducing physical damping materials at the camera mounts could isolate the sensors further and reduce the need for aggressive filtering.

Summary updated

Abstract

In this paper, we propose a visual-inertial framework able to efficiently estimate the camera poses of a non-rigid trinocular baseline for long-range depth estimation on-board a fast moving aerial platform. The estimation of the time-varying baseline is based on relative inertial measurements, a photometric relative pose optimizer, and a probabilistic wing model fused in an efficient Extended Kalman Filter (EKF) formulation. The estimated depth measurements can be integrated into a geo-referenced global map to render a reconstruction of the environment useful for local replanning algorithms. Based on extensive real-world experiments we describe the challenges and solutions for obtaining the probabilistic wing model, reliable relative inertial measurements, and vision-based relative pose updates and demonstrate the computational efficiency and robustness of the overall system under challenging conditions.
BibTeX
@misc{hinzmann2019flexible,
  author = {Timo Hinzmann and Cesar Cadena and Juan Nieto and Roland Siegwart},
  title = {{Flexible Trinocular: Non-rigid Multi-Camera-IMU Dense Reconstruction for UAV Navigation and Mapping}},
  year = {2019},
  eprint = {1908.08891},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1908.08891},
}
Figure