TL;DR
This paper introduces a novel framework for detecting and describing Foreign Object Debris (FOD) on airport pavements. By enhancing YOLOv7 for detection, integrating Segment Anything Model (SAM) for segmentation, and employing GPT for detailed, context-aware descriptions, the system significantly improves FOD identification and provides actionable insights for airport safety and maintenance.
What Problem Does This Paper Address?
Foreign Object Debris (FOD) on airport surfaces poses a critical safety hazard, potentially causing severe aircraft damage, operational disruptions, and substantial economic losses. Existing detection systems often struggle with small objects, challenging environmental conditions (like low light, glare, or rain), and lack the ability to provide interpretable, context-rich descriptions of the detected debris. This limits their utility for effective risk management and operational decision-making in complex airport environments. The paper aims to bridge this gap by developing an intelligent system that not only accurately detects FOD but also generates detailed semantic descriptions that inform maintenance and safety protocols.
Methods
The proposed framework, YOLO-SAM-GPT, is a multi-modal pipeline that integrates several key components:
- Enhanced YOLOv7-X for Small Object Detection: The base YOLOv7-X detector is augmented with:
- Coordinate Attention (CA): To improve the capture of spatial location information within channel attention maps, enhancing focus on small objects.
- Space-to-Depth Convolution (SPD-Conv): To preserve fine-grained spatial features during downsampling, crucial for small object detection.
- Gaussian Similarity IoU (GS-IoU) Loss: A novel loss function that approximates bounding boxes as Gaussian distributions, improving localization stability and accuracy, especially for small objects.
- Cascaded YOLO-SAM for Instance Segmentation: Detected bounding boxes from YOLOv7-X are used as prompts for the Segment Anything Model (SAM). This cascaded approach leverages high-quality masks generated by SAM to refine object localization and enable open-set segmentation (detecting previously unseen object types).
- Structured Prompt Engineering for GPT-Based Description: A sophisticated prompt engineering mechanism is employed to guide a large language model (like GPT) in generating detailed semantic descriptions. This involves:
- Spatial Encoding: Numbering and ordering detected objects based on visual reading habits (left-to-right, top-to-bottom).
- Contextual Scene Information: Extracting environmental details like ground material, lighting, spatial location, and surrounding facilities.
- Domain Knowledge Integration: Incorporating a semantic cue lexicon with FOD categories, material attributes, and risk levels to ensure domain-specific and safety-critical descriptions.
Datasets
The study utilizes a custom dataset of 12,000 annotated images of FOD collected from Luoyang Beijiao Airport and Xuzhou Guanyin Airport. These images capture a wide range of real-world conditions, including diverse weather (rain, fog, snow), lighting (daytime, nighttime, glare), and airport zones (runways, taxiways, aprons). The dataset is noted for its superior diversity, annotation granularity, and environmental realism compared to existing public FOD datasets. For evaluation of the description generation module, 500 images were selected as a test set, each with manually annotated environmental descriptions.
Key Insights and Ideas
- The synergy of detection, segmentation, and language generation is powerful: Combining these modalities unlocks a deeper understanding of FOD beyond simple localization.
- Small object detection is significantly boosted by spatial attention and specialized IoU: The integration of Coordinate Attention and GS-IoU proved critical for accurately identifying small debris, a common challenge.
- SAM's prompt-based segmentation is highly effective, but prompt quality matters: Using enlarged bounding boxes (150%) as prompts for SAM yielded superior segmentation results compared to center points or unadjusted boxes.
- Structured prompt engineering is key to generating useful, actionable descriptions: Simply feeding raw image data to LLMs is insufficient. Incorporating spatial ordering, environmental context, and domain-specific risk information dramatically improves the quality and relevance of generated descriptions.
- Airport FOD detection requires robustness across diverse conditions: The system demonstrates promising adaptability to challenging scenarios like complete darkness, low-light, high-glare, and rainy weather, though some limitations remain.
Key Results
- Detection Performance: The enhanced YOLOv7-X achieved an AP of 90.5%, a 4.9% improvement over the baseline, with a notable 4.5-6.0% increase in APs for small objects. Inference latency increased by only 1.7%.
- Segmentation Performance: The cascaded YOLO-SAM approach achieved high segmentation performance, with MPA of 96.88%, MIoU of 0.986, and F1 Score of 0.988 when using optimized bounding box prompts.
- Description Accuracy: The structured prompt engineering significantly boosted description accuracy from 76.0% to 91.3%, and prompt consistency from 61.2% to 89.6%.
- End-to-End Latency: The total system latency is 384.8 ms per image (approx. 2.6 FPS), which is suitable for semi-real-time or edge-assisted asynchronous deployment, prioritizing robustness and interpretability over strict real-time video requirements.
Limitations and Future Work
- Real-time Constraints: The current end-to-end latency of 384.8 ms per image does not meet strict real-time video requirements, limiting its use in high-frame-rate applications.
- Performance in Extreme Conditions: While robust, the system occasionally misses small or low-reflectivity debris in complete darkness and can produce false positives or inaccurate attribute descriptions in rainy conditions due to water-induced blur and reduced contrast.
- Future Work Suggestions:
- Improving robustness against detection errors through iterative refinement and feedback-based loops.
- Exploring complementary region proposal methods.
- Incorporating multi-modal inputs to mitigate failures in complete darkness or severe weather.
- Developing model compression and lightweight designs for deployment on low-resource airport environments and edge devices.
Summary updated