Rethinking Certification for Trustworthy Machine-Learning-Based Applications

Marco Anisetti, Claudio A. Ardagna, Nicola Bena, Ernesto Damiani

TL;DR

This paper argues that traditional certification methods are insufficient for modern machine learning (ML)-based applications due to their non-deterministic nature. They propose a novel multi-factor certification framework that considers data, training process, and the ML model itself, offering a path towards ensuring the trustworthiness of these complex systems.

What Problem Does This Paper Address?

The increasing reliance on Machine Learning (ML) in applications, especially those operating in cloud-edge environments, presents a significant challenge for ensuring trustworthiness. Unlike traditional deterministic software, ML models exhibit non-deterministic behavior, making it difficult to assess and guarantee critical non-functional properties like fairness, robustness, and privacy. Existing certification schemes, designed for deterministic systems, are ill-equipped to handle the inherent uncertainties of ML. This gap leaves users and regulators concerned about the safety and reliability of applications driven by ML, prompting a need for new assurance techniques.

Methods

The authors propose a multi-factor certification framework for ML-based applications. This framework extends traditional certification by breaking down the assessment into three independent but interconnected factors:

  • Factor Data ($f_d$): This assesses the dataset used for training and validation. It focuses on characteristics of the data itself, such as its balance and potential for poisoning, which can significantly impact the ML model's behavior and properties.
  • Factor Process ($f_p$): This examines the training process employed to develop the ML model. It considers aspects like the adoption of specific training techniques (e.g., boosting, transfer learning) and the integrity of the training pipeline, including the inspection of checkpoints.
  • Factor Model ($f_m$): This evaluates the ML model's behavior when it's in operation. This is crucial for understanding how the model performs in real-world scenarios and how it responds to various inputs.

Each factor has its own certification model, comprising a property definition, a target of certification, and an evidence collection model. The overall certification of an ML application is then derived from the assessments of these three factors.

Datasets

The paper uses a reference scenario of a malware detection application as a running example. This application is based on a Deep Learning model trained on real-world data and synthetic data generated by a Generative Adversarial Network (GAN). The paper does not introduce new datasets but rather uses this hypothetical scenario to illustrate the application of their proposed certification scheme.

Key Insights and Ideas

  • The "Groundhog Day" of Certification: The paper highlights that the challenges faced in certifying service-oriented applications in 2012 are resurfacing with ML, demanding a similar paradigm shift in assurance.
  • ML is Not Just Code: The trustworthiness of an ML application is not solely dependent on the code but also on the data it's trained on and the process used for training. This necessitates a broader scope for certification.
  • Interconnectedness of Factors: The failure in one factor (e.g., a poisoned training dataset) can directly lead to failures in another (e.g., a non-robust ML model), emphasizing the need for a holistic view.
  • Beyond Deterministic Properties: Traditional non-functional properties like confidentiality and integrity need to be re-evaluated and redefined in the context of ML, alongside entirely new ML-specific properties like explainability and robustness.

Key Results

The paper presents a conceptual framework and applies it to a hypothetical malware detection scenario. In this scenario, the proposed certification scheme evaluates the robustness of the application against inference-time attacks.

  • Factor Data ($f_d$): The assessment for the training set was positive (✔), indicating no poisoned samples were found.
  • Factor Process ($f_p$): The assessment for the training process was negative (X), as it lacked essential strengthening techniques like randomized smoothing and adversarial training.
  • Factor Model ($f_m$): The assessment for the ML model in operation was also negative (X), with a recall of 0 on adversarial samples, meaning the model misclassified all of them as benign.

The overall conclusion for the malware detector in this scenario is that it lacks the proper robustness to operate in adversarial settings, primarily due to issues in the training process.

Limitations and Future Work

The authors acknowledge that their work is a "first step" and that the road ahead is "long and impervious." Key limitations and future directions include:

  • Complementary Evidence Collection: The proposed evidence collection techniques need to be complemented by other approaches like abstract interpretation and inspection of surrogate white-box models.
  • Life Cycle Management: The life cycle of ML models and their certificates requires careful management, as ML models need continuous adaptation to evolving system behavior and data drift.
  • Driving ML Evolution: Certification should not just track changes but also actively drive the evolution of ML models to ensure they consistently support desired properties across all factors.
  • ML Pipeline Recursion (C4): The recursive nature of ML pipelines, where components can themselves be ML pipelines, presents a challenge that remains an open issue for future work.

Summary updated

BibTeX
@article{anisetti2023rethinking,
  author = {Marco Anisetti and Claudio A. Ardagna and Nicola Bena and Ernesto Damiani},
  title = {{Rethinking Certification for Trustworthy Machine-Learning-Based Applications}},
  journal = {IEEE Internet Comput},
  year = {2023},
  doi = {10.1109/mic.2023.3322327},
}
Figure