On Extending the Automatic Test Markup Language (ATML) for Machine Learning

Tyler Cody, Bingtong Li, Peter A. Beling

Abstract

This paper addresses the urgent need for messaging standards in the operational test and evaluation (T&E) of machine learning (ML) applications, particularly in edge ML applications embedded in systems like robots, satellites, and unmanned vehicles. It examines the suitability of the IEEE Standard 1671 (IEEE Std 1671), known as the Automatic Test Markup Language (ATML), an XML-based standard originally developed for electronic systems, for ML application testing. The paper explores extending IEEE Std 1671 to encompass the unique challenges of ML applications, including the use of datasets and dependencies on software. Through modeling various tests such as adversarial robustness and drift detection, this paper offers a framework adaptable to specific applications, suggesting that minor modifications to ATML might suffice to address the novelties of ML. This paper differentiates ATML's focus on testing from other ML standards like Predictive Model Markup Language (PMML) or Open Neural Network Exchange (ONNX), which concentrate on ML model specification. We conclude that ATML is a promising tool for effective, near real-time operational T&E of ML applications, an essential aspect of AI lifecycle management, safety, and governance.

Introduction

As machine learning (ML) transitions from controlled research environments to critical operational infrastructure—such as autonomous vehicles, industrial robotics, and satellite systems—the need for robust, standardized testing becomes paramount. While significant progress has been made in standardizing model architectures through formats like ONNX and PMML, a critical gap remains: the lack of a universal language to describe operational test and evaluation (T&E) requirements and results. This paper explores the adaptation of the IEEE Standard 1671, known as the Automatic Test Markup Language (ATML), to bridge this gap.

The Standardization Gap

Currently, testing for "edge ML" is often conducted in an ad-hoc manner. There is no standardized way to communicate test requirements (e.g., "this model must maintain 80% accuracy under adversarial attack") or results across disparate systems. While standards like ONNX describe how a model is built, they do not describe how it should be verified in the field. The authors argue that rather than reinventing the wheel, the industry can leverage and extend ATML—an XML-based language originally designed for electronic hardware—to handle the unique requirements of ML software.

Methodology: Mapping ML to Hardware Schemas

The study evaluates ATML's suitability by conducting a mapping exercise where core ML concepts are integrated into existing hardware-centric schemas:

  1. Unit Under Test (UUT): An ML model, such as a Convolutional Neural Network (CNN), is treated as the UUT. The UUTDescription schema is used to define the model's architecture, layers, and input/output shapes.
  2. Test Station: The computing environment—including specific versions of Python, TensorFlow, and hardware accelerators like the NVIDIA RTX 3090—is documented using the TestStationDescription schema. This ensures environmental reproducibility, which is a common challenge in ML deployment.

Modeling Operational Tests

The paper demonstrates the flexibility of ATML by providing XML-based modeling for three critical ML testing scenarios:

  • Cross-Validation: Defining specific data folds and performance thresholds for model validation.
  • Adversarial Testing: Specifying perturbation parameters (e.g., epsilon values) and required robustness scores to ensure the model can withstand intentional attacks.
  • Drift Detection: Monitoring statistical shifts in input data over time, a vital requirement for maintaining model integrity in dynamic edge environments where "concept drift" can degrade performance.

Addressing the Dataset Challenge

A primary limitation identified in the study is ATML’s lack of native support for large-scale datasets. While ATML excels at describing discrete signals (like a sine wave), it was not originally built to handle multi-gigabyte CSV or image databases. To address this, the authors propose a "Dataset ID" workaround: using TestRequirement elements as pointers or identifiers to external data stores. This allows the XML files to remain lightweight and interoperable while maintaining a clear link to the data payloads used during the test flow.

Key Insights and Results

This qualitative proof-of-concept confirms that ATML can effectively describe ML models as UUTs and standardize failure reporting through the TestResults schema. A key takeaway is the distinction between specification (what the model is) and testing (how the model performs). While ONNX allows for model sharing, ATML provides the necessary framework for model verification, safety, and governance.

The authors conclude that a brand-new standard is likely unnecessary. Instead, a new sub-standard (potentially an "IEEE 1671.X") focusing on software-dominant systems and data payloads would suffice to bring ML into the fold of standardized electronic testing.

Limitations and Future Work

As a modeling-first paper, the primary limitation is the lack of a full-scale physical implementation. Future research directions include:

  • Practical Implementation: Building an actual Automatic Test System (ATS) that utilizes these ATML schemas in conjunction with open-source ML tools.
  • Handling Complexity: Developing more granular ways to describe complex statistical distributions and multidimensional datasets within the ATML framework.
  • Integration: Exploring how ATML can interact with emerging AI governance tools, such as "model cards," to provide a comprehensive view of model health and compliance.

Summary updated

Abstract

This paper addresses the urgent need for messaging standards in the operational test and evaluation (T&E) of machine learning (ML) applications, particularly in edge ML applications embedded in systems like robots, satellites, and unmanned vehicles. It examines the suitability of the IEEE Standard 1671 (IEEE Std 1671), known as the Automatic Test Markup Language (ATML), an XML-based standard originally developed for electronic systems, for ML application testing. The paper explores extending IEEE Std 1671 to encompass the unique challenges of ML applications, including the use of datasets and dependencies on software. Through modeling various tests such as adversarial robustness and drift detection, this paper offers a framework adaptable to specific applications, suggesting that minor modifications to ATML might suffice to address the novelties of ML. This paper differentiates ATML's focus on testing from other ML standards like Predictive Model Markup Language (PMML) or Open Neural Network Exchange (ONNX), which concentrate on ML model specification. We conclude that ATML is a promising tool for effective, near real-time operational T&E of ML applications, an essential aspect of AI lifecycle management, safety, and governance.
BibTeX
@inproceedings{cody2024extending,
  author = {Tyler Cody and Bingtong Li and Peter A. Beling},
  title = {{On Extending the Automatic Test Markup Language (ATML) for Machine Learning}},
  booktitle = {SysCon},
  year = {2024},
  url = {https://openreview.net/forum?id=EIRUb7MwRC},
}
Figure