Intelligent Detection and Description of Foreign Object Debris on Airport Pavements via Enhanced YOLOv7 and GPT-Based Prompt Engineering

Hanglin Cheng, Ruoxi Zhang, Ruiheng Zhang, Yihao Li, Lei Yang, Weiguang ZhangSensors 2025 · DOI

TL;DR

This paper introduces a novel framework for detecting and describing Foreign Object Debris (FOD) on airport pavements. By enhancing YOLOv7 for detection, integrating Segment Anything Model (SAM) for segmentation, and employing GPT for detailed, context-aware descriptions, the system significantly improves FOD identification and provides actionable insights for airport safety and maintenance.

What Problem Does This Paper Address?

Foreign Object Debris (FOD) on airport surfaces poses a critical safety hazard, potentially causing severe aircraft damage, operational disruptions, and substantial economic losses. Existing detection systems often struggle with small objects, challenging environmental conditions (like low light, glare, or rain), and lack the ability to provide interpretable, context-rich descriptions of the detected debris. This limits their utility for effective risk management and operational decision-making in complex airport environments. The paper aims to bridge this gap by developing an intelligent system that not only accurately detects FOD but also generates detailed semantic descriptions that inform maintenance and safety protocols.

Methods

The proposed framework, YOLO-SAM-GPT, is a multi-modal pipeline that integrates several key components:

  • Enhanced YOLOv7-X for Small Object Detection: The base YOLOv7-X detector is augmented with:
  • Coordinate Attention (CA): To improve the capture of spatial location information within channel attention maps, enhancing focus on small objects.
  • Space-to-Depth Convolution (SPD-Conv): To preserve fine-grained spatial features during downsampling, crucial for small object detection.
  • Gaussian Similarity IoU (GS-IoU) Loss: A novel loss function that approximates bounding boxes as Gaussian distributions, improving localization stability and accuracy, especially for small objects.
  • Cascaded YOLO-SAM for Instance Segmentation: Detected bounding boxes from YOLOv7-X are used as prompts for the Segment Anything Model (SAM). This cascaded approach leverages high-quality masks generated by SAM to refine object localization and enable open-set segmentation (detecting previously unseen object types).
  • Structured Prompt Engineering for GPT-Based Description: A sophisticated prompt engineering mechanism is employed to guide a large language model (like GPT) in generating detailed semantic descriptions. This involves:
  • Spatial Encoding: Numbering and ordering detected objects based on visual reading habits (left-to-right, top-to-bottom).
  • Contextual Scene Information: Extracting environmental details like ground material, lighting, spatial location, and surrounding facilities.
  • Domain Knowledge Integration: Incorporating a semantic cue lexicon with FOD categories, material attributes, and risk levels to ensure domain-specific and safety-critical descriptions.

Datasets

The study utilizes a custom dataset of 12,000 annotated images of FOD collected from Luoyang Beijiao Airport and Xuzhou Guanyin Airport. These images capture a wide range of real-world conditions, including diverse weather (rain, fog, snow), lighting (daytime, nighttime, glare), and airport zones (runways, taxiways, aprons). The dataset is noted for its superior diversity, annotation granularity, and environmental realism compared to existing public FOD datasets. For evaluation of the description generation module, 500 images were selected as a test set, each with manually annotated environmental descriptions.

Key Insights and Ideas

  • The synergy of detection, segmentation, and language generation is powerful: Combining these modalities unlocks a deeper understanding of FOD beyond simple localization.
  • Small object detection is significantly boosted by spatial attention and specialized IoU: The integration of Coordinate Attention and GS-IoU proved critical for accurately identifying small debris, a common challenge.
  • SAM's prompt-based segmentation is highly effective, but prompt quality matters: Using enlarged bounding boxes (150%) as prompts for SAM yielded superior segmentation results compared to center points or unadjusted boxes.
  • Structured prompt engineering is key to generating useful, actionable descriptions: Simply feeding raw image data to LLMs is insufficient. Incorporating spatial ordering, environmental context, and domain-specific risk information dramatically improves the quality and relevance of generated descriptions.
  • Airport FOD detection requires robustness across diverse conditions: The system demonstrates promising adaptability to challenging scenarios like complete darkness, low-light, high-glare, and rainy weather, though some limitations remain.

Key Results

  • Detection Performance: The enhanced YOLOv7-X achieved an AP of 90.5%, a 4.9% improvement over the baseline, with a notable 4.5-6.0% increase in APs for small objects. Inference latency increased by only 1.7%.
  • Segmentation Performance: The cascaded YOLO-SAM approach achieved high segmentation performance, with MPA of 96.88%, MIoU of 0.986, and F1 Score of 0.988 when using optimized bounding box prompts.
  • Description Accuracy: The structured prompt engineering significantly boosted description accuracy from 76.0% to 91.3%, and prompt consistency from 61.2% to 89.6%.
  • End-to-End Latency: The total system latency is 384.8 ms per image (approx. 2.6 FPS), which is suitable for semi-real-time or edge-assisted asynchronous deployment, prioritizing robustness and interpretability over strict real-time video requirements.

Limitations and Future Work

  • Real-time Constraints: The current end-to-end latency of 384.8 ms per image does not meet strict real-time video requirements, limiting its use in high-frame-rate applications.
  • Performance in Extreme Conditions: While robust, the system occasionally misses small or low-reflectivity debris in complete darkness and can produce false positives or inaccurate attribute descriptions in rainy conditions due to water-induced blur and reduced contrast.
  • Future Work Suggestions:
    • Improving robustness against detection errors through iterative refinement and feedback-based loops.
  • Exploring complementary region proposal methods.
  • Incorporating multi-modal inputs to mitigate failures in complete darkness or severe weather.
  • Developing model compression and lightweight designs for deployment on low-resource airport environments and edge devices.

Summary updated

Abstract

Foreign Object Debris (FOD) on airport pavements poses a serious threat to aviation safety, making accurate detection and interpretable scene understanding crucial for operational risk management. This paper presents an integrated multi-modal framework that combines an enhanced YOLOv7-X detector, a cascaded YOLO-SAM segmentation module, and a structured prompt engineering mechanism to generate detailed semantic descriptions of detected FOD. Detection performance is improved through the integration of Coordinate Attention, Spatial-Depth Conversion (SPD-Conv), and a Gaussian Similarity IoU (GSIoU) loss, leading to a 3.9% gain in [email protected] for small objects with only a 1.7% increase in inference latency. The YOLO-SAM cascade leverages high-quality masks to guide structured prompt generation, which incorporates spatial encoding, material attributes, and operational risk cues, resulting in a substantial improvement in description accuracy from 76.0% to 91.3%. Extensive experiments on a dataset of 12,000 real airport images demonstrate competitive detection and segmentation performance compared to recent CNN- and transformer-based baselines while achieving robust semantic generalization in challenging scenarios, such as complete darkness, low-light, high-glare nighttime conditions, and rainy weather. A runtime breakdown shows that the enhanced YOLOv7-X requires 40.2 ms per image, SAM segmentation takes 142.5 ms, structured prompt construction adds 23.5 ms, and BLIP-2 description generation requires 178.6 ms, resulting in an end-to-end latency of 384.8 ms per image. Although this does not meet strict real-time video requirements, it is suitable for semi-real-time or edge-assisted asynchronous deployment, where detection robustness and semantic interpretability are prioritized over ultra-low latency. The proposed framework offers a practical, deployable solution for airport FOD monitoring, combining high-precision detection with context-aware description generation to support intelligent runway inspection and maintenance decision-making.
BibTeX
@inproceedings{cheng2025intelligent,
  author = {Hanglin Cheng and Ruoxi Zhang and Ruiheng Zhang and Yihao Li and Lei Yang and Weiguang Zhang},
  title = {{Intelligent Detection and Description of Foreign Object Debris on Airport Pavements via Enhanced YOLOv7 and GPT-Based Prompt Engineering}},
  booktitle = {Sensors},
  year = {2025},
  doi = {10.3390/s25165116},
}
Figure