Introduction
Integrating machine learning (ML) into safety-critical aeronautical systems presents significant challenges, primarily due to the industry's rigorous certification standards, such as DO-178C. Unlike traditional software verified through explicit logic, ML systems are data-driven, requiring a new paradigm for verification. A critical component of this is ensuring that training and testing datasets comprehensively cover the Operational Design Domain (ODD)—the specific environmental and operational conditions under which the system is intended to function safely.
This paper addresses the gap between high-level system requirements and actionable data specifications in the context of Vision-Based Landing (VBL). Specifically, it proposes a framework to translate flight physics constraints into verifiable image-level requirements for runway detection.
Methodology: From CONOPS to Image-Level ODD
The authors propose a multi-stage workflow to refine a system-level ODD into a dataset-level specification. This process ensures that every data point is traceable to the system's Concept of Operations (CONOPS).
The workflow involves three primary steps:
- ODD Refinement: Mapping the 3D "landing cone" (defined by flight physics) onto a 2D image plane using geometric projection matrices. This creates an "Image-level ODD."
- Concept Partitioning: Visual elements are categorized into Primary (essential features like runway shape), Secondary (reinforcing features like taxiways), Tertiary (potential biases like specific lighting), and Quaternary (false positive triggers like highways).
- Data Quality Requirements (DQRs): Establishing metrics for Source Suitability, Completeness, Representativeness, and Accuracy to validate the resulting dataset.
Geometric Ground Truth and the Landing Cone
In aviation, the ODD is fundamentally geometric. By defining a generic landing approach cone, the authors can mathematically predict the expected position of the runway in any given image. This geometric foundation allows for automated labeling and rigorous validation of the dataset's coverage.
Expert knowledge is integrated through a "Concept Filter." For example, while tire marks are useful visual cues, they are classified as "secondary" because relying on them could lead to failures on new runways. This distinction ensures the ML model focuses on robust, primary features.
The LARD Dataset
The framework was applied to develop the Landing Approach Runway Detection (LARD) dataset. LARD is a hybrid dataset combining synthetic imagery and real-world footage. Synthetic data from Microsoft Flight Simulator (MSFS) and Google Earth Studio is used to cover edge cases and precise geometric trajectories, while real-world footage provides ground-truth realism to validate the synthetic utility.
Results and Validation
The LARD dataset was successfully validated against the defined DQRs. Key findings include:
- Completeness: 3D visualizations confirmed that the training and test sets fully covered the required landing cone.
- Representativeness: Runway center positions were evenly distributed across the image space, preventing center-bias in the trained models.
- Synthetic Utility: High-fidelity simulations showed high similarity to real-world environments, suggesting their viability for certification-grade datasets where real data is scarce or hard to label.
Conclusion
This research provides a structured approach to dataset design for safety-critical ML. By bridging the gap between system-level ODDs and image-level requirements through geometric projection and expert concept partitioning, the authors offer a path toward the formal certification of AI in aviation. Future work will expand the ODD to include adverse weather and night conditions, and further investigate the "reality gap" between synthetic and real-world data.
Summary updated