How to design a dataset compliant with an ML-based system ODD?

Cyril Cappi, Noémie Cohen, Mélanie Ducoffe, Christophe Gabreau, Laurent Gardes, Adrien Gauffriau, Jean-Brice Ginestet, Franck Mamalet, Vincent Mussot, Claire Pagetti, David VigourouxarXiv 2024 · DOI · arXiv

Introduction

Integrating machine learning (ML) into safety-critical aeronautical systems presents significant challenges, primarily due to the industry's rigorous certification standards, such as DO-178C. Unlike traditional software verified through explicit logic, ML systems are data-driven, requiring a new paradigm for verification. A critical component of this is ensuring that training and testing datasets comprehensively cover the Operational Design Domain (ODD)—the specific environmental and operational conditions under which the system is intended to function safely.

This paper addresses the gap between high-level system requirements and actionable data specifications in the context of Vision-Based Landing (VBL). Specifically, it proposes a framework to translate flight physics constraints into verifiable image-level requirements for runway detection.

Methodology: From CONOPS to Image-Level ODD

The authors propose a multi-stage workflow to refine a system-level ODD into a dataset-level specification. This process ensures that every data point is traceable to the system's Concept of Operations (CONOPS).

The ODD Design Workflow illustrates the transition from high-level CONOPS to verifiable dataset requirements.

The workflow involves three primary steps:

  1. ODD Refinement: Mapping the 3D "landing cone" (defined by flight physics) onto a 2D image plane using geometric projection matrices. This creates an "Image-level ODD."
  2. Concept Partitioning: Visual elements are categorized into Primary (essential features like runway shape), Secondary (reinforcing features like taxiways), Tertiary (potential biases like specific lighting), and Quaternary (false positive triggers like highways).
  3. Data Quality Requirements (DQRs): Establishing metrics for Source Suitability, Completeness, Representativeness, and Accuracy to validate the resulting dataset.

Geometric Ground Truth and the Landing Cone

In aviation, the ODD is fundamentally geometric. By defining a generic landing approach cone, the authors can mathematically predict the expected position of the runway in any given image. This geometric foundation allows for automated labeling and rigorous validation of the dataset's coverage.

The geometric parameters used to define the landing cone, including lateral and vertical path angles.

Expert knowledge is integrated through a "Concept Filter." For example, while tire marks are useful visual cues, they are classified as "secondary" because relying on them could lead to failures on new runways. This distinction ensures the ML model focuses on robust, primary features.

The LARD Dataset

The framework was applied to develop the Landing Approach Runway Detection (LARD) dataset. LARD is a hybrid dataset combining synthetic imagery and real-world footage. Synthetic data from Microsoft Flight Simulator (MSFS) and Google Earth Studio is used to cover edge cases and precise geometric trajectories, while real-world footage provides ground-truth realism to validate the synthetic utility.

Results and Validation

The LARD dataset was successfully validated against the defined DQRs. Key findings include:

  • Completeness: 3D visualizations confirmed that the training and test sets fully covered the required landing cone.
  • Representativeness: Runway center positions were evenly distributed across the image space, preventing center-bias in the trained models.
  • Synthetic Utility: High-fidelity simulations showed high similarity to real-world environments, suggesting their viability for certification-grade datasets where real data is scarce or hard to label.

Conclusion

This research provides a structured approach to dataset design for safety-critical ML. By bridging the gap between system-level ODDs and image-level requirements through geometric projection and expert concept partitioning, the authors offer a path toward the formal certification of AI in aviation. Future work will expand the ODD to include adverse weather and night conditions, and further investigate the "reality gap" between synthetic and real-world data.

Summary updated

Abstract

This paper focuses on a Vision-based Landing task and presents the design and the validation of a dataset that would comply with the Operational Design Domain (ODD) of a Machine-Learning (ML) system. Relying on emerging certification standards, we describe the process for establishing ODDs at both the system and image levels. In the process, we present the translation of high-level system constraints into actionable image-level properties, allowing for the definition of verifiable Data Quality Requirements (DQRs). To illustrate this approach, we use the Landing Approach Runway Detection (LARD) dataset which combines synthetic imagery and real footage, and we focus on the steps required to verify the DQRs. The replicable framework presented in this paper addresses the challenges of designing a dataset compliant with the stringent needs of ML-based systems certification in safety-critical applications.
BibTeX
@misc{cappi2024how,
  author = {Cyril Cappi and Noémie Cohen and Mélanie Ducoffe and Christophe Gabreau and Laurent Gardes and Adrien Gauffriau and Jean-Brice Ginestet and Franck Mamalet and Vincent Mussot and Claire Pagetti and David Vigouroux},
  title = {{How to design a dataset compliant with an ML-based system ODD?}},
  year = {2024},
  eprint = {2406.14027},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2406.14027},
}
Figure