Trustworthy Machine Learning

Bálint Mucsányi, Michael Kirchhof, Elisa Nguyen, Alexander Rubinstein, Seong Joon OharXiv 2023 · DOI · arXiv

Abstract

This work formalizes the burgeoning field of Trustworthy Machine Learning (TML), advocating for a fundamental paradigm shift from "ML 1.0" (simple label prediction) to "ML 2.0" (understanding the underlying reasoning process). By synthesizing state-of-the-art research in Out-of-Distribution (OOD) generalization, explainability, and uncertainty quantification, the authors provide a unified framework for building AI systems that are reliable in real-world deployments. The work highlights the critical need for rigorous evaluation standards and a transition toward dataset-centric solutions to bridge the current "trustworthiness gap."

The cover of 'Trustworthy Machine Learning', outlining the core themes of Theory, Applications, and Intuitions.

Introduction: The Trustworthiness Gap

Despite the unprecedented success of deep learning across various domains, standard models often exhibit fragility when moved from the laboratory to the real world. These models are frequently prone to "shortcut bias," a phenomenon where they rely on spurious correlations—such as a specific background landscape—rather than causal features to make predictions. Furthermore, most high-performing models remain "black boxes," unable to explain their reasoning and often exhibiting overconfidence even when making incorrect predictions.

To address these issues, this work establishes three primary pillars of Trustworthy Machine Learning:

  1. OOD Generalization: Ensuring models maintain performance on data distributions not seen during training.
  2. Explainability: Making the decision-making process transparent and interpretable to human users.
  3. Uncertainty Quantification: Enabling models to recognize the limits of their own knowledge.

The Paradigm Shift: From ML 1.0 to ML 2.0

A central thesis of this work is the transition to "ML 2.0." While traditional machine learning focuses on learning the conditional probability $P(Y|X)$ (predicting a label $Y$ given input $X$), the authors argue that trustworthy systems must learn $P(X, Z, Y)$. Here, $Z$ represents the "how"—the intermediate cues, reasoning steps, and causal factors that lead to a conclusion. By explicitly modeling the reasoning process, we can move away from models that achieve high accuracy through brittle shortcuts.

Core Pillars of Trustworthiness

Out-of-Distribution (OOD) Generalization

To combat domain-specific biases, the authors discuss methods like Domain-Adversarial Training (DANN) and Group Distributionally Robust Optimization (Group DRO). These techniques aim to force models to ignore superficial domain features and focus on invariant, causal characteristics. However, the work notes a significant challenge: "Information Leakage." Many current OOD results are inadvertently inflated because researchers use information from the test domain (such as hyperparameter tuning) during the development phase.

Explainability and the Soundness Trade-off

The text evaluates various post-hoc attribution methods, including Saliency Maps, Integrated Gradients, LIME, and SHAP. A key insight provided is the inherent tension between an explanation being "sound" (perfectly faithful to the model's internal complexity) and "interpretable" (simple enough for a human to grasp). To address this, the authors introduce Class Activation Latent Mapping (CALM), a probabilistic approach designed to provide well-calibrated explanations by construction.

Uncertainty Quantification

Trustworthy models must "know when they don't know." The work covers Bayesian approaches such as Deep Ensembles and MC Dropout, as well as deterministic distance-based methods like Mahalanobis distance. These methods help separate aleatoric uncertainty (inherent noise in the data) from epistemic uncertainty (uncertainty due to the model's lack of knowledge).

The work is structured as a textbook, utilizing clear pedagogical elements such as blue-boxed definitions to introduce core concepts.

Empirical Findings and Evaluation

The authors utilize a wide array of benchmarks, including ImageNet-C, PACS, DomainBed, and CIFAR-10H, to rigorously test these methods. Several key results emerge:

  • Simple Calibration: Temperature Scaling is identified as a remarkably effective and simple fix for model overconfidence, often reducing Expected Calibration Error (ECE) from ~10% to under 2%.
  • Ensembles vs. Dropout: Deep Ensembles consistently outperform MC Dropout in both accuracy and the quality of uncertainty estimates on large-scale benchmarks like ImageNet.
  • The Power of Baselines: In many OOD benchmarks (such as DomainBed), a well-tuned Empirical Risk Minimization (ERM) baseline often performs as well as, or better than, highly complex specialized algorithms, suggesting that algorithmic complexity does not always equate to better generalization.
  • Sanity Checks: The paper highlights "scandals" in explainability research where complex attribution methods were found to be no more informative than simple edge detectors when subjected to rigorous sanity checks like the Cascading Randomization test.

Future Directions: A Dataset-Centric Approach

The authors conclude that the next major leap in TML will likely not come from increasingly complex algorithms, but from "data hunting." This involves collecting interventional data and human-provided "how" signals—such as mouse tracks or rationales during annotation—to provide the $Z$ labels needed for ML 2.0. Additionally, there is a pressing need for Human-in-the-Loop (HITL) research to ensure that model explanations and uncertainty estimates actually improve the decision-making of human experts, such as clinicians or legal professionals.

Summary updated

Abstract

As machine learning technology gets applied to actual products and solutions, new challenges have emerged. Models unexpectedly fail to generalize to small changes in the distribution, tend to be confident on novel data they have never seen, or cannot communicate the rationale behind their decisions effectively with the end users. Collectively, we face a trustworthiness issue with the current machine learning technology. This textbook on Trustworthy Machine Learning (TML) covers a theoretical and technical background of four key topics in TML: Out-of-Distribution Generalization, Explainability, Uncertainty Quantification, and Evaluation of Trustworthiness. We discuss important classical and contemporary research papers of the aforementioned fields and uncover and connect their underlying intuitions. The book evolved from the homonymous course at the University of T\"ubingen, first offered in the Winter Semester of 2022/23. It is meant to be a stand-alone product accompanied by code snippets and various pointers to further sources on topics of TML. The dedicated website of the book is https://trustworthyml.io/.
BibTeX
@misc{mucsanyi2023trustworthy,
  author = {Bálint Mucsányi and Michael Kirchhof and Elisa Nguyen and Alexander Rubinstein and Seong Joon Oh},
  title = {{Trustworthy Machine Learning}},
  year = {2023},
  eprint = {2310.08215},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2310.08215},
}
Figure