Can we trust AI? towards practical implementation and theoretical analysis in trustworthy machine learning

Kaidi Xu2021 · DOI

Introduction

Deep Neural Networks (DNNs) have achieved extraordinary performance in domains ranging from medical image analysis to natural language processing. However, their deployment in security-critical applications is hindered by a fundamental flaw: vulnerability to adversarial attacks. These attacks involve adding sophisticated, often imperceptible perturbations to benign inputs, causing the model to misclassify them with high confidence. To build truly trustworthy machine learning systems, we must understand these vulnerabilities through both empirical experimentation and rigorous theoretical analysis.

This research addresses the "Trustworthy Machine Learning" gap by developing novel methods to both break and verify DNNs. The work spans interpretable digital attacks, robust physical-world evasions, and high-speed automated verification frameworks.

Empirical Investigations: Exposing Vulnerabilities

Traditional adversarial attacks often perturb every pixel in an image, making it difficult to discern why a model failed. This dissertation introduces StrAttack (Structured Attack), a framework that explores group sparsity in perturbations. By using a sliding mask and the Alternating Direction Method of Multipliers (ADMM), StrAttack identifies the "minimally sufficient" spatial structures required to cause misclassification. This approach bridges the gap between attacking a model and explaining its internal logic, revealing the specific features—often irrelevant textures—the AI over-relies on.

Moving beyond the digital realm, the research tackles the challenge of Physical World Fragility. Most digital attacks fail in the real world due to lighting shifts, distance variations, and object deformation. To address this, the author developed the Adversarial T-shirt. Unlike rigid adversarial patches, this T-shirt is designed to evade person detectors (like YOLOv2) even as the fabric wrinkles and folds on a moving person. This was achieved by combining Expectation over Transformation (EoT) with Thin Plate Spline (TPS) mapping to model non-rigid deformations. In physical testing, this method achieved a 57% success rate, significantly outperforming the 18% success rate of standard adversarial patches.

Theoretical Foundations: Proving Safety

While empirical attacks expose flaws, theoretical frameworks are necessary to guarantee safety. The research extends robustness to Graph Neural Networks (GNNs), which are used in social networks and chemical modeling. Because graph structures are discrete (edges either exist or they do not), the author utilized convex relaxation to enable gradient-based adversarial training, creating the first optimization-based defense for GNNs.

At the core of the dissertation's theoretical contribution is LiRPA (Linear Relaxation based Perturbation Analysis). LiRPA is an automated framework that computes provable linear bounds for neural network outputs under input perturbations. Much like automatic differentiation simplified backpropagation, LiRPA simplifies the process of calculating safety bounds for complex architectures.

To make these theoretical bounds practical, the research introduces Fast and Complete Verification. Historically, proving a model's safety was an NP-hard problem restricted to small models due to the slow nature of CPU-based Linear Programming (LP) solvers. By formulating LiRPA bounds as matrix operations, the author leveraged the parallel processing power of GPUs. Combined with a Branch and Bound (BaB) algorithm, this framework accelerates verification by 10x to 30x over previous state-of-the-art methods, enabling the verification of larger models like WideResNet and datasets like Tiny ImageNet for the first time.

Conclusion and Future Outlook

The path to trustworthy AI requires a dual focus on understanding how models break and proving when they are safe. This research demonstrates that sparsity can lead to interpretability, physical deformations can be modeled mathematically, and formal verification can be scaled through GPU acceleration.

Future work will focus on Patch Transformers to better model environmental noise and the development of real-time verification systems. The ultimate goal is to provide safety guarantees in milliseconds, a requirement for the safe integration of AI into autonomous vehicles and other high-stakes, real-time environments.

Summary updated

Abstract

Deep learning or deep neural networks (DNNs) have achieved extraordinary performance in many application domains such as image classification, object detection and recognition, natural language processing and medical image analysis. It has been well accepted that DNNs are vulnerable to adversarial attacks, which raises concerns of DNNs in security-critical applications and may result in disastrous consequences. Adversarial attacks are usually implemented by generating adversarial examples, i.e., adding sophisticated perturbations onto benign examples, such that adversarial examples are classified by the DNN as target (wrong) labels instead of the correct labels of the benign examples. The adversarial machine learning aims to study this phenomenon and leverage it to build robust machine learning systems and explain DNNs. In this dissertation, we present the mechanism of adversarial machine learning in both empirical and theoretical ways. Specifically, we first introduce a uniform adversarial attack generation framework, structured attack (StrAttack), which explores group sparsity in adversarial perturbations by sliding a mask through images aiming for extracting key spatial structures. Second, we discuss the feasibility of adversarial attack in the physical world and introduce a powerful framework, Expectation over Transformation (EoT). Utilize EoT with Thin Plate Spline (TPS) transformation, we can generate Adversarial T-shirts, a robust physical adversarial example for evading person detectors even if it could undergo non-rigid deformation due to a moving person's pose changes. Third, we stand on the defense side and propose the first adversarial training method based on Graph Neural Network. Fourth, we introduce Linear relaxation based perturbation analysis (LiRPA) for neural networks, which computes the provable linear bounds of output neurons given a certain amount of input perturbation. LiRPA studies the adversarial example in a theoretical way and can guarantee the test accuracy of a model by given perturbation constraints. Finally, leveraging the efficient LiRPA with branch and bound, we speed up the conventional Linear Programming-based complete verification framework by an order of magnitude. In the future, we plan to study on a novel patch transformer network to truthfully model real-world physical transformations empirically. In addition, at the formal robustness direction, we plan to explore the complete verification in real-time, that given sufficient time, the verifier should give a definite "yes/no" answer for a property under verification efficiently. Our LiRPA framework combined with GPUs can accelerate this procedure potentially.--Author's abstract
BibTeX
@misc{xu2021can,
  author = {Kaidi Xu},
  title = {{Can we trust AI? towards practical implementation and theoretical analysis in trustworthy machine learning}},
  year = {2021},
  doi = {10.17760/d20413930},
}
Figure