On Extending the Automatic Test Markup Language (ATML) for Machine Learning

Tyler Cody, Bingtong Li, Peter A. Beling

TL;DR

This paper explores how to adapt the Automatic Test Markup Language (ATML)—a long-standing IEEE standard for hardware testing—to the world of machine learning (ML). The authors demonstrate that with minor extensions, ATML can provide a much-needed standardized framework for the operational test and evaluation (T&E) of ML models, particularly for "edge" applications like robots and satellites.

What Problem Does This Paper Address?

As machine learning moves from research labs into critical infrastructure (autonomous vehicles, industrial robots, satellites), we face a major hurdle: standardization. While we have standards like ONNX or PMML to describe how a model is built, we lack a universal language to describe how a model should be tested in the field.

Currently, testing for "edge ML" is often ad-hoc. There is no standard way to communicate test requirements (e.g., "this model must maintain 80% accuracy under adversarial attack") or results across different systems. The authors argue that we don't need to reinvent the wheel; instead, we can extend IEEE Std 1671 (ATML), an XML-based language originally designed for electronic hardware, to handle the unique quirks of ML software.

Methods

The authors conduct a "suitability study" by attempting to model common ML testing scenarios using existing ATML schemas. Their approach involves:

  1. Mapping ML Concepts to Hardware Schemas: They treat an ML model as a "Unit Under Test" (UUT) and the computing environment (Python, TensorFlow, GPUs) as a "Test Station."
  2. Modeling Specific Tests: They provide XML code examples for three critical ML tests:
  3. Cross-Validation: Defining folds and performance thresholds.
  4. Adversarial Testing: Specifying perturbation sizes (epsilon) and required robustness scores.
  5. Drift Detection: Monitoring statistical changes in input data over time.
  6. Identifying Gaps: They pinpoint where ATML fails—specifically, its lack of native support for datasets. While ATML is great at describing a sine wave signal, it isn't built to handle a multi-gigabyte CSV or image database.

Datasets

The paper does not introduce a new benchmark dataset or perform quantitative experiments on specific data. Instead, it uses conceptual data references (e.g., DataSet_123) within its XML examples to demonstrate how an Automatic Test System would incorporate dataset payloads into a standardized test flow.

Key Insights and Ideas

  • Testing vs. Specification: A major "aha" moment is the distinction between standards like ONNX (which describe the model's architecture) and ATML (which describes the test's intent). You can use ONNX to share a model, but you need ATML to verify it.
  • The "Dataset ID" Workaround: Since ATML doesn't natively "speak" dataset, the authors propose using custom TestRequirement elements that act as pointers (identifiers) to external data stores. This maintains interoperability without bloating the XML files.
  • Minor vs. Major Changes: The authors conclude that a brand-new standard isn't necessary. A new sub-standard (perhaps an "IEEE 1671.X") that focuses on software-dominant systems and data payloads would likely suffice.
  • Edge ML Focus: The paper emphasizes that standardized messaging is most critical for edge devices, where models must be monitored for "concept drift" or degradation in real-time without human intervention.

Key Results

The paper provides a qualitative proof-of-concept. It successfully demonstrates that:

  • ML Models as UUTs: A Convolutional Neural Network (CNN) can be effectively described using ATML’s UUTDescription schema, including its layers and input shapes.
  • Environment Standardization: The TestStationDescription can successfully document the specific versions of Python, TensorFlow, and hardware (like an NVIDIA RTX 3090) used during a test, ensuring reproducibility.
  • Failure Reporting: The TestResults schema can be used to generate standardized diagnostic messages when a model fails a cross-validation or robustness check.

Limitations and Future Work

The primary limitation is that this is a modeling-first paper. The authors have mapped out the "grammar" of how ML testing should look in ATML, but they haven't yet built a full-scale physical implementation.

Future directions include: * Practical Implementation: Building an actual Automatic Test System (ATS) that uses these ATML schemas in conjunction with open-source ML tools. * Handling Complexity: Developing better ways to describe complex statistical distributions and multidimensional datasets within the ATML framework, as current hardware-centric languages (SDL and TDL) struggle with the granularity required for ML. * Integration: Exploring how ATML can interact more deeply with "model cards" and other emerging AI governance tools.

Summary updated

Abstract

This paper addresses the urgent need for messaging standards in the operational test and evaluation (T&E) of machine learning (ML) applications, particularly in edge ML applications embedded in systems like robots, satellites, and unmanned vehicles. It examines the suitability of the IEEE Standard 1671 (IEEE Std 1671), known as the Automatic Test Markup Language (ATML), an XML-based standard originally developed for electronic systems, for ML application testing. The paper explores extending IEEE Std 1671 to encompass the unique challenges of ML applications, including the use of datasets and dependencies on software. Through modeling various tests such as adversarial robustness and drift detection, this paper offers a framework adaptable to specific applications, suggesting that minor modifications to ATML might suffice to address the novelties of ML. This paper differentiates ATML’s focus on testing from other ML standards like Predictive Model Markup Language (PMML) or Open Neural Network Exchange (ONNX), which concentrate on ML model specification. We conclude that ATML is a promising tool for effective, near real-time operational T&E of ML applications, an essential aspect of AI lifecycle management, safety, and governance.
BibTeX
@inproceedings{cody2024extending,
  author = {Tyler Cody and Bingtong Li and Peter A. Beling},
  title = {{On Extending the Automatic Test Markup Language (ATML) for Machine Learning}},
  booktitle = {SysCon},
  year = {2024},
  url = {https://openreview.net/forum?id=EIRUb7MwRC},
}

Loading related content...

Figure