The Care Label Concept: A Certification Suite for Trustworthy and Resource-Aware Machine Learning

Katharina Morik, Helena Kotthaus, Lukas Heppe, Danny Heinrich, Raphael Fischer, Andreas Pauly, Nico PiatkowskiarXiv 2021 · OpenReview

Abstract

Machine learning applications have become ubiquitous. This has led to an increased effort of making machine learning trustworthy. Explainable and fair AI have already matured. They address knowledgeable users and application engineers. For those who do not want to invest time into understanding the method or the learned model, we offer care labels: easy to understand at a glance, allowing for method or model comparisons, and, at the same time, scientifically well-based. On one hand, this transforms descriptions as given by, e.g., Fact Sheets or Model Cards, into a form that is well-suited for end-users. On the other hand, care labels are the result of a certification suite that tests whether stated guarantees hold. In this paper, we present two experiments with our certification suite. One shows the care labels for configurations of Markov random fields (MRFs). Based on the underlying theory of MRFs, each choice leads to its specific rating of static properties like, e.g., expressivity and reliability. In addition, the implementation is tested and resource consumption is measured yielding dynamic properties. This two-level procedure is followed by another experiment certifying deep neural network (DNN) models. There, we draw the static properties from the literature on a particular model and data set. At the second level, experiments are generated that deliver measurements of robustness against certain attacks. We illustrate this by ResNet-18 and MobileNetV3 applied to ImageNet.

1. Introduction

As machine learning (ML) moves from research labs into everyday applications, "trustworthiness" has become a critical bottleneck. While expert-level tools like Model Cards or Fact Sheets exist, they are often too dense for end-users or application engineers who need to compare models quickly. There is also a frequent gap between theoretical guarantees and practical implementation. A model might have low theoretical complexity on paper but prove energy-hungry or fragile when deployed on specific hardware. To bridge this gap, we propose "Care Labels": easy-to-understand, scientifically grounded summaries that allow for rapid comparison of ML methods and models.

2. The Certification Framework

The proposed certification suite transforms technical descriptions into a standardized format. It processes three primary inputs: theoretical foundations, the specific code implementation, and benchmark data.

The Certification Suite workflow: Theory, implementation, and benchmark data are processed to generate standardized Care Labels.

The framework operates on two levels:

  • Static Properties (Expert Knowledge): Derived from scientific literature and theoretical bounds, these include Expressivity (modeling capability), Reliability (theoretical guarantees), and Usability.
  • Dynamic Properties (Execution Environment): Measured by executing the model in a specific environment. This includes Resource Consumption (energy in kWh, memory in GB, runtime in ms) and Robustness (performance under corruption, perturbation, and noise).

3. Rating Methodology and Automated Verification

To make these technical metrics accessible, the suite generates a visual rating from A to D. This process involves mapping complex criteria to discrete grades based on established rules and resource benchmarks.

The logic for deriving A-D ratings from static criteria (like complexity classes) and dynamic measurements.

For example, theoretical runtime and memory complexity are evaluated using Big-O notation. The suite performs "Complexity Class Checks" to ensure the implementation's scaling behavior matches the mathematical theory. Dynamic ratings are currently quantile-based, comparing a model's performance against a database of peers to determine its relative standing.

4. Experimental Evaluation

We evaluated the certification suite using two distinct ML domains to demonstrate its versatility.

4.1 Deep Neural Networks (DNNs)

Using the ImageNet dataset, we compared architectures including AlexNet, VGG11, ResNet-18, and MobileNetV3. A key finding was the "Efficiency Paradox": MobileNetV3, while efficient for inference (receiving an 'A'), received a 'D' for training energy because it required significantly more epochs to reach target accuracy, resulting in a higher carbon footprint. Furthermore, while ResNet-18 showed high accuracy, its Care Label revealed 'C' and 'D' ratings for robustness against noise and corruption, highlighting potential risks in safety-critical deployments.

4.2 Markov Random Fields (MRFs)

We tested inference algorithms like Junction Tree (JT) and Loopy Belief Propagation (LBP) on synthetic grid graphs. The labels clearly illustrated the inherent trade-offs: JT received an 'A' for Reliability (mathematically exact) but a 'D' for Resources (exponential scaling), whereas LBP received a 'B' for Resources but a 'D' for Reliability (approximation with no guarantees).

5. Conclusion

The Care Label concept provides a necessary bridge between technical complexity and user trust. By combining theoretical "expert knowledge" with empirical "execution environment" testing, the certification suite offers a holistic view of model performance. Future work will focus on expanding the scope to domains like Natural Language Processing (NLP) and Reinforcement Learning, as well as refining the rating system as the model database grows.

Summary updated

Abstract

Machine learning applications have become ubiquitous. This has led to an increased effort of making machine learning trustworthy. Explainable and fair AI have already matured. They address knowledgeable users and application engineers. For those who do not want to invest time into understanding the method or the learned model, we offer care labels: easy to understand at a glance, allowing for method or model comparisons, and, at the same time, scientifically well-based. On one hand, this transforms descriptions as given by, e.g., Fact Sheets or Model Cards, into a form that is well-suited for end-users. On the other hand, care labels are the result of a certification suite that tests whether stated guarantees hold. In this paper, we present two experiments with our certification suite. One shows the care labels for configurations of Markov random fields (MRFs). Based on the underlying theory of MRFs, each choice leads to its specific rating of static properties like, e.g., expressivity and reliability. In addition, the implementation is tested and resource consumption is measured yielding dynamic properties. This two-level procedure is followed by another experiment certifying deep neural network (DNN) models. There, we draw the static properties from the literature on a particular model and data set. At the second level, experiments are generated that deliver measurements of robustness against certain attacks. We illustrate this by ResNet-18 and MobileNetV3 applied to ImageNet.
BibTeX
@misc{morik2021care,
  author = {Katharina Morik and Helena Kotthaus and Lukas Heppe and Danny Heinrich and Raphael Fischer and Andreas Pauly and Nico Piatkowski},
  title = {{The Care Label Concept: A Certification Suite for Trustworthy and Resource-Aware Machine Learning}},
  year = {2021},
  eprint = {2106.00512},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2106.00512},
}
Figure