Vision-Based Landing Guidance through Tracking and Orientation Estimation

João P. K. Ferreira, João P. Pinto, Júlia Moura, Yi Li, Cristiano L. Castro, Plamen AngelovWACV 2025

TL;DR

This paper introduces a novel vision-based landing guidance system for fixed-wing aircraft that uses a state-of-the-art tracking model (LoRAT) to detect and follow runways in real-time. By generating synthetic landing videos and employing a robust keypoint extraction and pose estimation pipeline, the system offers a cost-effective and potentially more reliable alternative to traditional instrument landing systems.

What Problem Does This Paper Address?

Traditional automated landing systems for aircraft, such as Instrument Landing Systems (ILS) and Precision Approach Radar (PAR), are expensive to install and maintain, and their reliance on ground or satellite signals makes them vulnerable to interference. While vision-based systems offer a promising alternative, they face challenges in accurately detecting and tracking runways, especially under varying conditions and from a distance. This paper aims to bridge this gap by developing a robust and cost-effective vision-based landing guidance system that can accurately estimate an aircraft's pose relative to the runway.

Methods

The proposed framework consists of four main stages:

  1. Data Generation: To overcome the scarcity of aerial front-view landing videos, the authors leverage the generator code from the LARD (Landing Approach Runway Detection dataset) to create high-quality synthetic landing sequences. This allows for the generation of large datasets with automatic annotations.
  2. Runway Tracking (LoRAT): Instead of traditional object detection, the paper employs LoRAT (Low-Rank Adaptation of Transformers), a parameter-efficient fine-tuning method for vision transformers. LoRAT is used to track the runway within bounding boxes across video frames. This approach is chosen because the runway is a static object in the landing sequence, allowing the tracker to focus on a smaller region once the runway is initially detected.
  3. Keypoint Extraction (YOLOv8n-pose): To obtain precise landmarks for pose estimation, the authors use YOLOv8n-pose, a model designed for pose estimation tasks. To improve accuracy and efficiency, the images are cropped around the bounding box predicted by the LoRAT tracker before being fed into YOLOv8n-pose.
  4. Camera Orientation Estimation (PnP Algorithm): Finally, the Perspective-n-Point (PnP) algorithm is used to estimate the aircraft's relative pose (orientation and slant distance) to the runway. This is achieved by mapping the 3D coordinates of runway corner points to their 2D projections in the image, using the keypoints extracted by YOLOv8n-pose.

Datasets

  • LARD (Landing Approach Runway Detection dataset): Used for generating synthetic landing sequences for training the tracking model. The authors specifically utilized the generator code to create smooth landing videos.
  • Custom-generated synthetic landing sequences: The authors created training sequences of approximately 500 frames and testing sequences of approximately 35 frames from the LARD dataset. They also used a subset of the LARD dataset for evaluating the orientation estimation.

Key Insights and Ideas

  • Tracking over Detection for Runways: The paper argues that for runway detection in landing sequences, tracking is more efficient and robust than traditional object detection. Once the runway is in view, it remains the primary object of interest, allowing the tracker to focus its search area and improve accuracy.
  • Synthetic Data Generation is Crucial: The lack of readily available aerial front-view landing video sequences was a significant hurdle. The authors' use of the LARD generator to create synthetic data was essential for training their deep learning models, particularly the LoRAT tracker.
  • Cropping for Keypoint Accuracy: Cropping the image around the tracked runway bounding box before keypoint extraction significantly improved the performance of the YOLOv8n-pose model, highlighting the importance of focusing the model's attention on the relevant region.
  • PnP Sensitivity: While the PnP algorithm performed well, the authors noted its susceptibility to minor deviations in keypoint positions, suggesting avenues for future improvement.

Key Results

  • Landing Runway Tracking: The LoRAT tracker achieved strong performance, with a Success Score (SUC) of 80.1% and Precision of 88.1% when the initial bounding box was close to the ground truth. Performance degraded significantly with larger displacements of the initial bounding box.
  • Keypoint Extraction: Training YOLOv8n-pose on cropped images resulted in superior performance across all evaluated metrics (Precision, Recall, mAP50, mAP50-95) compared to training on full-sized images. For example, Precision increased from 95.45% to 98.92% with cropping.
  • Orientation Estimation: The PnP algorithm, using detected keypoints, achieved mean orientation errors close to zero degrees across various scenarios, comparable to using ground truth keypoints. The roll angle showed the highest standard deviation, indicating sensitivity to keypoint variations. Slant distance estimation also showed promising results, generally improving as the aircraft approached the runway.

Limitations and Future Work

  • Initial Bounding Box Sensitivity: The tracking performance is sensitive to the accuracy of the initial bounding box provided by the object detector. Significant deviations can lead to decreased performance.
  • PnP Sensitivity to Keypoint Deviations: The PnP algorithm's accuracy is affected by minor inaccuracies in the detected keypoint positions.
  • Future Work:
    • Increase the number of uncorrelated keypoints provided to the PnP algorithm.
  • Enrich keypoint estimates by modifying the LoRAT tracker to predict keypoint displacements or by performing sensor fusion (e.g., combining visual estimates with onboard Inertial Navigation System (INS) data using a filtering technique like the Extended Kalman Filter).
  • Extend experiments to more challenging landing scenarios (maneuvers, shakiness, real footage).
  • Test the approach on other public datasets (FS2020, RLD) and compare with state-of-the-art methods.

Summary updated

BibTeX
@inproceedings{ferreira2025vision,
  author = {João P. K. Ferreira and João P. Pinto and Júlia Moura and Yi Li and Cristiano L. Castro and Plamen Angelov},
  title = {{Vision-Based Landing Guidance through Tracking and Orientation Estimation}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2025},
}
Figure